:W&BᾺQôIM amvz3ch1ª; 9 /0.28.1+bc2cf24b1bd9fa3a1b6cbeb920087b832afc526f0.70.0zŸõÜ€‚" 2fj9heg8chw5 amvz3ch1ŠÜ amvz3ch1escheuller-uc-san-diego spatop-sweep" _wandb‚{}  batch_size‚1024  dropout‚0.20021280877836  hidden_dim‚128 ( learning_rate‚0.00021284925886345515  num_embedding_layers‚10Bdpfx_blocks_amvz3ch1box8or7c7j$dp-spatop-sweepagent-blocks-v2-fb276Š ú´ôÒà溹š3 (H¨¸hp"3.12.10*0.28.1j linux-x86_64µ*Õ?; amvz3ch1*  _wandb‚ {"runtime":0}  _runtime‚0S=5`ÒÚ -Linux-6.8.0-124-generic-x86_64-with-glibc2.36CPython 3.12.10 ú´ôÒðøð„*-of*>/data/spatop/options_files/sweep_fixed_dp/config_amvz3ch1.json*-l* /scratch/logs*-n*dpfx_blocks_amvz3ch12-m spanet.trainRdanielprimosch@gmail.comZ /scratch/logsb$dp-spatop-sweepagent-blocks-v2-fb276r/opt/conda/bin/pythonº hho5p2ndp35uae1p9nf90jq1uzayl1lxæPò2  requirements.txtPM§W amvz3ch1ZF (H¨¸ (H¨¸ (hpð"3.12.10*0.28.1j linux-x86_64”/g\Y amvz3ch1ZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.28.1j linux-x86_64|Ò.}* amvz3ch1b trainer/global_step2oMu\  amvz3ch1ZK (H¨¸ (H¨¸(8hpð"3.12.10*0.28.1j linux-x86_64sîûª/  amvz3ch1b*"trainer/global_step2¿‰TLK  amvz3ch1j: û´ôÒ¨ŠñÛ*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] ûú“Ç ÒÁ€@ˆ@’NVIDIA A100 80GB PCIe˜¢ /€€€€ý/€€Ö½Û$ª€Àû”ÌÂS NVIDIA A100 80GB PCIe€€€€À€6"Ampere*(GPU-ed01d6ad-b673-30ac-3501-763022903663Ê13.2º hho5p2ndp35uae1p9nf90jq1uzayl1lxUV“(“  amvz3ch1j  û´ôÒèæ„üî â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 866 K │ train │ 0 │ │ 1 │ encoder │ JetEncoderWithPairwise │ 1.3 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 19.0 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 12 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ Ws—®\ amvz3ch1jÊ û´ôÒЖËü·Trainable params: 21.2 M Non-trainable params: 50 Total params: 21.2 M Total estimated model params size (MB): 84.816 Modules in train mode: 2326 Modules in eval mode: 0 Total FLOPs: 0 wó@q amvz3ch1j` û´ôÒ¥ÛüPwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). YCÍ{ amvz3ch1jj û´ôÒ íÝüZwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). ¨PŒ5q amvz3ch1j` û´ôÒ çßüPwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). JªÔøt amvz3ch1jc û´ôÒȪáüSwandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). ž„n amvz3ch1j] û´ôÒØõâüMwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). £‚êì. amvz3ch1*œ  transformer_dim‚128  transformer_dim_scale‚2.0  initial_embedding_dim‚128  position_embedding_dim‚128  num_encoder_layers‚4 ! num_branch_embedding_layers‚3  num_branch_encoder_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1  num_attention_heads‚8 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚1  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚"all"  pairwise_block_embeddings‚1  pairwise_cross_type‚true  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 K event_info_file‚7"/data/spatop/event_files/v11/tt_hadronic_v7_full.yaml" F training_file‚4"/data/spatop/tpm70_wpm30_FB350_fixed/all_merged.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_limit‚1.0  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 epochs‚50 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output""Ƙ ë amvz3ch1jÙ û´ôÒÈŽŽÿÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ˜B“”N amvz3ch1j¼ û´ôÒøÑœÿ«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. è®*Ùê amvz3ch1jØ ý´ôÒèÈÒÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. N† O amvz3ch1j½ ý´ôÒð¿â­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. S[Äô‚(:’ ŠµôÒ׆Ì) disk./.usagePercent‚76.52632740184123 disk.rbd0.out‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0 network.sent‚32607% disk./.usageGB‚1174.8704414367676 disk.dm-0.in‚0 disk.dm-1.in‚0" disk.nvme0n1.out‚ 6.29150390625 disk.nvme1n1.in‚0 disk.md0.in‚0# memory_percent‚99.2962121963501 network.recv‚38889 disk.dm-0.out‚6.140625 disk.nvme0c0n1.in‚0" disk.nvme1n1.out‚ 0.75830078125( proc.memory.availableMB‚ 115.30859375 cpu‚25.633035968559213 proc.cpu.threads‚17 disk.dm-1.out‚ 0.177734375 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 6.29150390625$ disk.nvme1c1n1.out‚ 0.75830078125 disk.md0.out‚0.34375# proc.memory.rssMB‚ 2778.40234375* proc.memory.percent‚16.958022117614746\ËL‚(:“ ŠµôÒÍïÉÌ gpu.0.gpu‚82.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚43.0 gpu.0.powerWatts‚110.21' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚36.736666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0..m3b/ lr-AdamW/pg1"trainer/global_step2:JP€œÝ*3b/ lr-AdamW/pg2"trainer/global_step2:JPªø1ý-) _wandbruntime‚25  _runtime‚25õ¼&°Á¼ ' lr-AdamW/pg1‚1.3020741178912985e-05 ' lr-AdamW/pg2‚1.3020741178912985e-05 trainer/global_step‚49 % _timestamp‚1.7844865494149346e+09 _runtime‚ 25.98994104 _step‚0j¢ê¸Ã¾ trainer/global_step‚49 % _timestamp‚1.7844865494149346e+09 _runtime‚ 25.98994104 _step‚0 ' lr-AdamW/pg1‚1.3020741178912985e-05 ' lr-AdamW/pg2‚1.3020741178912985e-05µ¿i be BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JPàö r!bn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JPÂdúb"b^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JP×bøÝj#bf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JP{áœg$bc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JP¬p2×i%be BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JP+Dœƒn&bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JP&/îch'bd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JPfRoAh(bd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JPô´l)bh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPnúe—v*br ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JPÿ=ž“e+ba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPQ‰n,bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JPãù:Zg-bc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JPaÃ>q.bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JP›’¶g/bc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JP+kª\r0bn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JP?+F¤n1bj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JPƒX™^2bZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JP1©Éàr3bn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPzµy‡l4bh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JP$¢Ltl5bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JPs`© n6bj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JP¯ÖR h7bd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JP*H—mf8bb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JPº7i9be BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JPßÉýc:b_ èl;bh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JPݨÇsbt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPã˜s?bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP¯g,g@bc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JP kújiAbe BDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak"trainer/global_step2:JPUñÝUjBbf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JPn“;rpCbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current"trainer/global_step2:JP°Ï: mDbi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JPp÷8'nEbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPÍ—C^cFb_ DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JP[K÷ºpLbl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JPÁî£ylMbh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JP-¡û–aNb] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JPç¼ÙnoObk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JP ·›ÓiPbe BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JPoÂefQbb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JPè*~%kRbg DDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JP6ˆnOdSb` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JPÿ!lTbh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JP^µÂÌhUbd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JP ¬<#iVbe BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JP‰ ÉYtWbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP‰4TÐaXb] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JPþÉ`Yb\ 9DeviceStatsMonitor.on_train_batch_start/segment.all.freed"trainer/global_step2:JPßi@hZbd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JP?&ñn[bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JPoÎfc\b_ "–6o`bk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JPŽ`¹nabj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JP-ÓS¥pbbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JP¼©!kcbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JPÞôqndbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JPɘÙÒlebh EDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current"trainer/global_step2:JP;^_•`fb\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JP?,uÑmgbi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JP^nØÚfhbb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JPå˜ dib` =DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak"trainer/global_step2:JP³×çejba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JPÙ‘ú8jkbf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JP{x!slbo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP5Ù%`mmbi FDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated"trainer/global_step2:JP…ÂKTonbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JPäù‡lnobj GDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated"trainer/global_step2:JPQ&«kpbg DDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated"trainer/global_step2:JPÃ>…Ösqbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JP¢z¬ srbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JPqœ;nsbj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPÃÁlåWtbS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JP¼Ã½ãkubg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JPøsV-kvbg DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated"trainer/global_step2:JP€[,Xgwbc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JP•CO¤vxbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JPE´šfybb ?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed"trainer/global_step2:JP=l–›qzbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JP,À-?b{b^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JP:]ë_|b[ 8DeviceStatsMonitor.on_train_batch_start/segment.all.peak"trainer/global_step2:JP½„;vk}bg DDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JP¦*½e~ba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JP1Œ.6qbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JPÈ,ð™p€bk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JP½ó*Òobj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JPÀ ‚Éi‚bd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JPý×”kƒbf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JPÜ>Ì#g„bb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JPÊúxj…be BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JP¢uÍE^†bY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JPˆ½˜ï`‡b[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JP{Z™â_ˆbZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JPq¨‹âq‰bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPw[æoŠbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JP hÂf‹ba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JPòæiÎpŒbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JPù½Žqbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JP¼“OjŽbe BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JPL|¹gbb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JPBä÷lbg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JP üdcq‘bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JPÈ+}Fu’bp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP@¹ÏMn“bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JPf8üDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537999215 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340577792 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚382925 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11965 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚128083 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 3736119805560 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 3729576441344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 3735762844742 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 GDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11894 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 3729593959936 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 46595700182 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 46680724992 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 3776257166336 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 3776615262720 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 48826353152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340577792 GDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚379082 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 3689167144560 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚379084 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚383205 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 3689184183920 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 46935621640 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚255120 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122892800 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339921458 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538661376 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 3729576441344 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚502563 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚379082 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P>DeviceStatsMonitorÄúy«ä.on_train_batch_start/active_bytes.all.freed‚ 3776257166336 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 46680724992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 47021302784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 2977696224768 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368t˜×–‹'…O E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11965 _step‚1 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚502563 K@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚881647 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚127805 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2252556800 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537999215 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340577792 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚382925 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11965 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚128083 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 3736119805560 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 3729576441344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 3735762844742 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 GDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11894 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 3729593959936 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 46595700182 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 46680724992 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 3776257166336 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 3776615262720 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 48826353152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340577792 GDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚379082 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 3689167144560 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚379084 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚383205 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 3689184183920 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 46935621640 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚255120 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122892800 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339921458 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538661376 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 3729576441344 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚502563 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚379082 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 3776257166336 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 46680724992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 47021302784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 2977696224768 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214285312 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239503 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358096384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3729593959936 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3026522577920 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 48949245952 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368ðXkižbd ADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated"trainer/global_step2:JP­˜¦vjŸbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JP7”m bh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JP‘ \Wh¡bc @DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current"trainer/global_step2:JPxd@“f¢ba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP¹ Ûâw£br ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPq3-m¤bh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JPóîße¥b` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JPA¾{ók¦bf CDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JP Mîq§bl IDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JPM’Òk¨bf CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JPlBç˜u©bp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JPæå/qªbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JPV!Ä4e«b` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JP’Ks'a¬b\ 9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak"trainer/global_step2:JPÁÉ·ˆm­bh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPË´›„j®be BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JP>B‘Ro¯bj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JP7¨Ñm°bh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JP›:Ä=n±bi FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JP ”ü]²bX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JPäWPf³ba >DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JPoº¼r´bm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JPzÕÅxoµbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JPÂëHd¶b_ IlÃbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JP‰M±iÄbd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JP+ÞAjiÅbd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JPwV!rÆbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JPÂîy_ÇbZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JP^Y¡ˆgÈbb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JP>·odÉb_ DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JPrç[ieÙb` =DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak"trainer/global_step2:JP.m„ÄfÚba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JPÈåøhpÛbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JPO‰é^kÜbf CDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current"trainer/global_step2:JP·˜ƒ¤mÝbh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JP»»ålÞbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JP*ÄgVßbQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JP1ĵlàbg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JPIžd¹báb] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JP„þ¤ümâbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JP§ã•cãb^ ;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak"trainer/global_step2:JPŠö‚läbg DDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JPÍ,'Ê`åb[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JPGÞñpæbk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JPP®Ñûhçbc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JPnXÑÑjèbe BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated"trainer/global_step2:JPzt§×gébb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JP3SýGwêbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPüÐöbiëbd ADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed"trainer/global_step2:JPªì 'gìbb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JPûêfÏbíb] :DeviceStatsMonitor.on_train_batch_end/allocation.all.freed"trainer/global_step2:JPbNôhîbc @DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak"trainer/global_step2:JPÍFU´eïb` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JPR€²uðbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JPYk~Zsñbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPYN¾ó^òbY 6DeviceStatsMonitor.on_train_batch_end/num_device_alloc"trainer/global_step2:JPng'egóbb ?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current"trainer/global_step2:JP£€¶nôbi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JP/Ýmõbh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JPãzéföba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JPršøøf÷ba >DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPÞD^pøbk HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JP9Åì„iùbd ADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated"trainer/global_step2:JPn™Ñ·eúb` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JPž°kSpûbk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JPªkólübg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JPëyië`ýb[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JP DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JP@%ò¯h‰bc @DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak"trainer/global_step2:JPÞ_ëoŠbj GDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current"trainer/global_step2:JP‹,Ñh‹bc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JPˆi h]ŒbX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JPÕÑxmdb_ DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JP-áE0^˜bY 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak"trainer/global_step2:JP¯à°*.™) _wandbruntime‚26  _runtime‚26û¨ñ3š‹M TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149407232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3762610266400 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚899038 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚512452 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚899038 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214286336 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 3086357365760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3036596981760 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚502702 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 3851378846720 E:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚889286 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚817 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 3036572557312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3762593227040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 3086506772992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339921570 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 trainer/global_step‚49 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚502702 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚297 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚473 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 47523923418 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 3803768210432 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚386586 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 49784808448 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚295 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 EÌ^;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚390481 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚386586 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 % _timestamp‚1.7844865497712054e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 49909791232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537999223 _step‚2 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚386584 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 47610636288 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚260203 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2252556800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124982784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚260201 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚390778 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 3803785729024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664‘´Ã0•&›M J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚295 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚390481 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚386586 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 % _timestamp‚1.7844865497712054e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 49909791232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537999223 _step‚2 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚386584 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 47610636288 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚260203 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2252556800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124982784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚260201 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚390778 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 3803785729024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149407232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3762610266400 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚899038 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚512452 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚899038 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214286336 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 3086357365760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3036596981760 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚502702 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 3851378846720 E:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚889286 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚817 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 3036572557312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3762593227040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 3086506772992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339921570 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 trainer/global_step‚49 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚502702 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚297 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚473 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 47523923418 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 3803768210432 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚386586 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 49784808448 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552wÍÔ^@œb; loss/FBt2/detection_loss"trainer/global_step2:JP¿ï-b( epoch"trainer/global_step2:JPs‰ÏËCžb> loss/SRqqt2/assignment_loss"trainer/global_step2:JP'hK]BŸb= loss/SRqqt2/detection_loss"trainer/global_step2:JP‰[x@ b; loss/FBt1/detection_loss"trainer/global_step2:JP´ÕO‰A¡b< loss/FRt2/assignment_loss"trainer/global_step2:JP[]ÉÌ@¢b; loss/FRt1/detection_loss"trainer/global_step2:JPó@ãÖ@£b; loss/FRt2/detection_loss"trainer/global_step2:JP°¯ç7¤b2 loss/total_loss"trainer/global_step2:JPqE*A¥b< loss/FRt1/assignment_loss"trainer/global_step2:JP€‹öpC¦b> loss/SRqqt1/assignment_loss"trainer/global_step2:JPÇK`A§b< loss/FBt1/assignment_loss"trainer/global_step2:JPš^ŽA¨b< loss/FBt2/assignment_loss"trainer/global_step2:JPòøåB©b= loss/SRqqt1/detection_loss"trainer/global_step2:JPè{._.ª) _wandbruntime‚26  _runtime‚26€M¥›ô«î /loss/FRt2/assignment_loss‚4.579547554555566 .loss/FRt1/detection_loss‚1.822656040915421 /loss/FRt2/detection_loss‚0.9182856554221926 # _timestamp‚1.78448654977345e+09 _runtime‚ 26.349297302 _step‚3 $loss/total_loss‚65.9264978555835 /loss/FRt1/assignment_loss‚5.367978436606271 1loss/SRqqt1/assignment_loss‚4.228192838205808 0loss/FBt1/assignment_loss‚2.2005055144026473 0loss/FBt2/assignment_loss‚2.2082688437567817 0loss/SRqqt1/detection_loss‚2.194156870137954 /loss/FBt2/detection_loss‚15.813953756789367 epoch‚0 trainer/global_step‚49 1loss/SRqqt2/assignment_loss‚4.178449940325609 1loss/SRqqt2/detection_loss‚3.3111744225395854 -loss/FBt1/detection_loss‚19.1033279819263L£ýqø¬ò 1loss/SRqqt2/assignment_loss‚4.178449940325609 1loss/SRqqt2/detection_loss‚3.3111744225395854 -loss/FBt1/detection_loss‚19.1033279819263 /loss/FRt2/assignment_loss‚4.579547554555566 .loss/FRt1/detection_loss‚1.822656040915421 /loss/FRt2/detection_loss‚0.9182856554221926 # _timestamp‚1.78448654977345e+09 _runtime‚ 26.349297302 _step‚3 $loss/total_loss‚65.9264978555835 /loss/FRt1/assignment_loss‚5.367978436606271 1loss/SRqqt1/assignment_loss‚4.228192838205808 0loss/FBt1/assignment_loss‚2.2005055144026473 0loss/FBt2/assignment_loss‚2.2082688437567817 0loss/SRqqt1/detection_loss‚2.194156870137954 /loss/FBt2/detection_loss‚15.813953756789367 epoch‚0 trainer/global_step‚49¤<)n­‚(:‘ ™µôÒòÛôË network.recv‚42427% disk./.usageGB‚1174.8705024719238 disk.md0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 1.22802734375) disk./.usagePercent‚76.52633137742542 disk.dm-0.out‚ 7.23486328125 disk.dm-1.out‚ 0.216796875" disk.nvme1n1.out‚ 1.22802734375" proc.memory.rssMB‚ 2778.4609375 disk.dm-0.in‚0! disk.nvme0c0n1.out‚ 7.59765625 disk.nvme0n1.out‚ 7.59765625 disk.nvme1n1.in‚0$ memory_percent‚99.30167198181152( proc.memory.percent‚16.9583797454834 network.sent‚41230 disk.md0.out‚ 0.63671875 disk.dm-1.in‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 114.4140625 cpu‚25.752947922036018 proc.cpu.threads‚17` Zž®‚(:“ ™µôÒŠ¡µÌ gpu.0.gpu‚88.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚48.0 gpu.0.powerWatts‚257.006' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.66866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.~—"¯‚(:— ¨µôÒê«êË) disk./.usagePercent‚76.52636914547521 disk.nvme0n1.in‚0$ disk.nvme1c1n1.out‚ 536.728515625" disk.nvme1n1.out‚ 536.728515625 disk.md0.out‚1.125 disk.dm-0.out‚ 8.23095703125( proc.memory.availableMB‚ 114.52734375) proc.memory.percent‚16.95847511291504 network.sent‚95164% disk./.usageGB‚1174.8710823059082 disk.dm-1.in‚0$ disk.nvme0c0n1.out‚ 9.04638671875 disk.dm-0.in‚0$ memory_percent‚99.30098056793213" proc.memory.rssMB‚ 2778.4765625 proc.cpu.threads‚17 network.recv‚48913 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0" disk.nvme0n1.out‚ 9.04638671875 disk.md0.in‚0 cpu‚25.769246699767585 disk.dm-1.out‚535.1240234375 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0Bµœ°‚(:‘ ¨µôÒÐð¯Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚49.0 gpu.0.powerWatts‚190.408' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚63.46933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨ó¯§.±) _wandbruntime‚50  _runtime‚50Çø»Â²¼ ' lr-AdamW/pg1‚2.6307211769640524e-05 ' lr-AdamW/pg2‚2.6307211769640524e-05 trainer/global_step‚99 $ _timestamp‚1.784486573525914e+09 _runtime‚ 50.101652917 _step‚4c~¤Æ³À $ _timestamp‚1.784486573525914e+09 _runtime‚ 50.101652917 _step‚4 ' lr-AdamW/pg1‚2.6307211769640524e-05 ' lr-AdamW/pg2‚2.6307211769640524e-05 trainer/global_step‚99pòç.´) _wandbruntime‚50  _runtime‚50Öå‰Í‹'µ…O QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 6018496288256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚988429 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚754182 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6018344776704 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 93038765190 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151511552 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 5921537276416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538662400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214286336 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚99 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356960818 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7532731538944 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7453866994568 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚754182 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7439182414336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 141784576 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚754184 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚998165 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7360488307920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214286336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 7439164895744 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚827 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚254483 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358096384 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127087104 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1742611 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358096384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7439182414336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7439164895744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 93378686648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340577792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 % _timestamp‚1.7844865735266845e+09 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7532373442560 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1752349 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚254775 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 5921512851968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 93549124608 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 93208546816 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 96959011840 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537999223 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚509172 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚763653 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 _runtime‚ 50.101718357 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 93208546816 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 7532731538944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340577792 _step‚5 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2252556800 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 96831924736 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7453510033750 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7360471268560 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 93549124608 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚988429 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚998165 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚483 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚754184 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚763947 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239511 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339921458 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738üºÉ϶‰O [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537999223 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚509172 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚763653 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 _runtime‚ 50.101718357 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 93208546816 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 7532731538944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340577792 _step‚5 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2252556800 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 96831924736 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7453510033750 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7360471268560 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 93549124608 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚988429 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚998165 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚483 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚754184 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚763947 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239511 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339921458 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 6018496288256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚988429 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚754182 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6018344776704 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 93038765190 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151511552 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 5921537276416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538662400 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214286336 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚99 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356960818 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7532731538944 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7453866994568 @8De¹…^ÀviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚754182 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7439182414336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 141784576 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚754184 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚998165 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7360488307920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214286336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 7439164895744 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚827 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚254483 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358096384 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 127087104 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1742611 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358096384 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7439182414336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7439164895744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 93378686648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340577792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 % _timestamp‚1.7844865735266845e+09 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7532373442560 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1752349 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚254775 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 5921512851968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 93549124608 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 93208546816 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 96959011840 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 7607495122944 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚514251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356960930 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7433914390400 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚771606 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2252556800 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 141784576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 5980413608960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚761686 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚301 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 7513374183424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1769740 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358103552 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340584960 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚299 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7433897351040 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7513356664832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538662400 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7513356664832 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚761684 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1008054 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚771305 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1769740 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 94138458112 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 6078178978304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 NDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214286336[-³Ù›&¹•M Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 7607495122944 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚514251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356960930 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7433914390400 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚771606 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2252556800 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 141784576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 5980413608960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚761686 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚301 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 7513374183424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1769740 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358103552 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340584960 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚299 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7433897351040 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7513356664832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538662400 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7513356664832 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚761684 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1008054 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚771305 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1769740 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 94138458112 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 6078178978304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 NDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214286336 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 94138458112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚998304 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 94479043072 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚257353 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 97918970880 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚761686 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259šÜ‚„.º) _wandbruntime‚50  _runtime‚50†³‚°÷»ñ 1loss/SRqqt1/detection_loss‚1.5390259268605055 0loss/SRqqt2/detection_loss‚5.368765260791406 %loss/total_loss‚66.12260513238925 epoch‚0 trainer/global_step‚99 % _timestamp‚1.7844865738981962e+09 1loss/SRqqt2/assignment_loss‚4.173050599172711 .loss/FBt1/detection_loss‚26.73187097391257 0loss/FBt1/assignment_loss‚1.8445605543943553 .loss/FRt1/detection_loss‚1.444378155036063 0loss/FRt2/detection_loss‚0.36635239101832906 .loss/FBt2/detection_loss‚8.740432596859238 _runtime‚ 50.473375735 /loss/FRt1/assignment_loss‚5.349413006215156 /loss/FRt2/assignment_loss‚4.512024301307397 1loss/SRqqt1/assignment_loss‚4.255293319823076 /loss/FBt2/assignment_loss‚1.797438046998448 _step‚7=nâqû¼õ 1loss/SRqqt2/assignment_loss‚4.173050599172711 .loss/FBt1/detection_loss‚26.73187097391257 0loss/FBt1/assignment_loss‚1.8445605543943553 .loss/FRt1/detection_loss‚1.444378155036063 0loss/FRt2/detection_loss‚0.36635239101832906 .loss/FBt2/detection_loss‚8.740432596859238 _runtime‚ 50.473375735 /loss/FRt1/assignment_loss‚5.349413006215156 /loss/FRt2/assignment_loss‚4.512024301307397 1loss/SRqqt1/assignment_loss‚4.255293319823076 /loss/FBt2/assignment_loss‚1.797438046998448 _step‚7 1loss/SRqqt1/detection_loss‚1.5390259268605055 0loss/SRqqt2/detection_loss‚5.368765260791406 %loss/total_loss‚66.12260513238925 epoch‚0 trainer/global_step‚99 % _timestamp‚1.7844865738981962e+09‹ ñ*½‚(:Ÿ ·µôÒùž“Ì disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0& disk.nvme1c1n1.out‚537.28662109375$ disk.nvme1n1.out‚537.28662109375% disk./.usageGB‚1174.8711166381836 disk.dm-1.in‚0$ disk.nvme0c0n1.out‚ 11.0810546875 disk.nvme0n1.in‚0 disk.nvme1n1.in‚0& proc.memory.availableMB‚ 114.171875) proc.memory.percent‚16.95857048034668 cpu‚25.761722305779024 network.recv‚52435" disk.nvme0n1.out‚ 11.0810546875 disk.nvme1c1n1.in‚0" proc.memory.rssMB‚ 2778.4921875 proc.cpu.threads‚17 network.sent‚112128) disk./.usagePercent‚76.52637138174131 disk.md0.in‚0 disk.md0.out‚ 1.3203125$ memory_percent‚99.30315017700195 disk.dm-0.in‚0 disk.dm-0.out‚10.32861328125 disk.dm-1.out‚535.1904296875驳—œ¾‚(:‘ ·µôÒêîÉÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚50.0 gpu.0.powerWatts‚243.827' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚81.27566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I(˜Ž.¿) _wandbruntime‚74  _runtime‚74ÖƒÐhÃÀ½ trainer/global_step‚149 $ _timestamp‚1.784486598147208e+09 _runtime‚ 74.722178091 _step‚8 ' lr-AdamW/pg1‚3.9593682360368064e-05 ' lr-AdamW/pg2‚3.9593682360368064e-05˜†AÇÁÁ ' lr-AdamW/pg2‚3.9593682360368064e-05 trainer/global_step‚149 $ _timestamp‚1.784486598147208e+09 _runtime‚ 74.722178091 _step‚8 ' lr-AdamW/pg1‚3.9593682360368064e-05»’D.Â) _wandbruntime‚74  _runtime‚74W Š²Ã¤O C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340577792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8865329479168 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 144972326912 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1493767 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358096384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1145049 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚381829 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131281408 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚11171257222758 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚763222 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11288489718784 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚11148753350144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538662400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1129284 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155705856 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚11031775392560 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 trainer/global_step‚149 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚382122 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11031792431920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 142405120 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339921458 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pooøÔâøl.freed‚0 _runtime‚ 74.722986496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 140076946432 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2252556800 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1129284 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 9010170524672 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 139481830198 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537999223 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2613313 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2623051 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356960818 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚11148753350144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚483 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚11288847815168 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 144841045504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 139736368640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214286336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9010326230528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214286336 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239511 % _timestamp‚1.7844865981481097e+09 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚11288847815168 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 139821751656 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1129282 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚11148770868736 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1484031 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538662400 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 139736368640 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720öùc ®'ĨO K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1129282 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚11148770868736 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1484031 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538662400 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 139736368640 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340577792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8865329479168 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 144972326912 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1493767 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358096384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1145049 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚381829 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131281408 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚11171257222758 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚763222 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11288489718784 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚11148753350144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538662400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1129284 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155705856 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚11031775392560 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 trainer/global_step‚149 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚382122 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11031792431920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 142405120 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339921458 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 74.722986496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 140076946432 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2252556800 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1129284 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 9010170524672 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 139481830198 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537999223 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2613313 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2623051 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356960818 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚11148753350144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚483 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚11288847815168 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 144841045504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 139736368640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214286336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9010326230528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214286336 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239511 % _timestamp‚1.7844865981481097e+09 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚11288847815168 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 139821751656 –%ÑàžÅ‚(:“ ƵôÒªÌÙË gpu.0.gpu‚68.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚52.0 gpu.0.powerWatts‚241.844' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚80.61466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã…”+Æ‚(:  ƵôÒýëèË% disk.nvme1c1n1.out‚538.1513671875# disk.nvme1n1.out‚538.1513671875 disk.md0.out‚ 1.72265625# memory_percent‚99.3098497390747) proc.memory.percent‚16.95859432220459 proc.cpu.threads‚17) disk./.usagePercent‚76.51089691723661 disk.rbd0.out‚0% disk.nvme0c0n1.out‚12.72119140625 disk.nvme1n1.in‚0 disk.md0.in‚0( proc.memory.availableMB‚ 113.07421875 network.sent‚158096 disk.dm-0.out‚ 11.6162109375! disk.dm-1.out‚535.55419921875 disk.rbd0.in‚0# disk.nvme0n1.out‚12.72119140625 disk.nvme1c1n1.in‚0# proc.memory.rssMB‚ 2778.49609375 cpu‚25.81618445480256 network.recv‚56307 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0$ disk./.usageGB‚1174.633544921875 disk.nvme0n1.in‚0s—L§.Ç) _wandbruntime‚75  _runtime‚75¼6µ&ȯM VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚11363969502720 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2640442 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214286336 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1136786 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 140666279936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2630690 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1503656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1493906 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356960930 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 140749975004 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8924230236160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚11245611528474 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339921570 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚11222962637824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 145932745216 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11222945119232 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358103552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 142405120 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11105201475040 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 _step‚10 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 145790985216 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538662400 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚384629 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141760000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1152639 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚11363611399168 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚384338 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚827 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚149 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚768303 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 140410053434 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1136784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9070162981376 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1503656 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 141006864896 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 % _timestamp‚1.7844865985052164e+09 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2630690 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2640442 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11105218514400 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0­y”ø}ɳM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚11363611399168 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚384338 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚827 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚149 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚768303 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 140410053434 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1136784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9070162981376 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1503656 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 141006864896 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 % _timestamp‚1.7844865985052164e+09 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2630690 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2640442 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11105218514400 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚11363969502720 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2640442 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all7@$Y<.peak‚ 12214286336 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1136786 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 140666279936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2630690 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1503656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1493906 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356960930 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 140749975004 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8924230236160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚11245611528474 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339921570 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚11222962637824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 145932745216 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11222945119232 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358103552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 142405120 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11105201475040 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 _step‚10 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 145790985216 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538662400 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚384629 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141760000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1152639 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 kE÷û.Ê) _wandbruntime‚75  _runtime‚75°Š3ûËõ /loss/FRt1/assignment_loss‚5.339204016304514 .loss/FBt1/detection_loss‚38.32204530015588 _step‚11 2loss/SRqqt2/assignment_loss‚3.9151133638344064 1loss/SRqqt1/detection_loss‚1.3620353059215082 0loss/SRqqt2/detection_loss‚4.392363544726214 _runtime‚ 75.081296606 /loss/FRt2/assignment_loss‚4.565597240330458 2loss/SRqqt1/assignment_loss‚4.0530373946178795 %loss/total_loss‚73.78512953486602 trainer/global_step‚149 % _timestamp‚1.7844865985065403e+09 epoch‚0 0loss/FBt1/assignment_loss‚0.8047447784079446 0loss/FBt2/assignment_loss‚0.4167778640985489 /loss/FRt1/detection_loss‚1.3844103045158536 /loss/FRt2/detection_loss‚0.3817282749748533 -loss/FBt2/detection_loss‚8.848072146977966ÉöÿÌù /loss/FRt2/assignment_loss‚4.565597240330458 2loss/SRqqt1/assignment_loss‚4.0530373946178795 %loss/total_loss‚73.78512953486602 trainer/global_step‚149 % _timestamp‚1.7844865985065403e+09 epoch‚0 0loss/FBt1/assignment_loss‚0.8047447784079446 0loss/FBt2/assignment_loss‚0.4167778640985489 /loss/FRt1/detection_loss‚1.3844103045158536 /loss/FRt2/detection_loss‚0.3817282749748533 -loss/FBt2/detection_loss‚8.84807214697796 /loss/FRt1/assignment_loss‚5.339204016304514 .loss/FBt1/detection_loss‚38.32204530015588 _step‚11 2loss/SRqqt2/assignment_loss‚3.9151133638344064 1loss/SRqqt1/detection_loss‚1.3620353059215082 0loss/SRqqt2/detection_loss‚4.392363544726214 _runtime‚ 75.081296606 ú›À+Í‚(:  ÕµôÒ¨áãË disk.dm-0.out‚ 18.0654296875 disk.dm-1.in‚0 disk.rbd0.out‚0$ disk.nvme1c1n1.out‚ 539.455078125" disk.nvme1n1.out‚ 539.455078125 network.sent‚194504 disk.dm-0.in‚0 disk.nvme0c0n1.in‚0% disk.nvme0c0n1.out‚18.90771484375 disk.nvme0n1.in‚0 disk.nvme1n1.in‚0# memory_percent‚99.3093729019165( proc.memory.availableMB‚ 113.15234375 network.recv‚60336 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0# proc.memory.rssMB‚ 2778.49609375) proc.memory.percent‚16.95859432220459 proc.cpu.threads‚17! disk.dm-1.out‚535.69091796875# disk.nvme0n1.out‚18.90771484375 disk.md0.in‚0 disk.md0.out‚ 2.15234375 cpu‚25.804244090451093) disk./.usagePercent‚76.51097021707007% disk./.usageGB‚1174.6346702575684ÊR{‘΂(:† ÕµôÒ°ÞœÌ gpu.0.gpu‚89.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚52.0 gpu.0.powerWatts‚244.17' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚81.39 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šNN<.Ï) _wandbruntime‚98  _runtime‚98‡uÁл % lr-AdamW/pg1‚5.28801529510956e-05 % lr-AdamW/pg2‚5.28801529510956e-05 trainer/global_step‚199 % _timestamp‚1.7844866219566345e+09 _runtime‚ 98.533665768 _step‚12QVz9ÅÑ¿ % lr-AdamW/pg1‚5.28801529510956e-05 % lr-AdamW/pg2‚5.28801529510956e-05 trainer/global_step‚199 % _timestamp‚1.7844866219566345e+09 _runtime‚ 98.533665768 _step‚12 Ì?9Y.Ò) _wandbruntime‚98  _runtime‚98™%8±'Ó«O SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1989369 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 142405120 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1504382 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1979633 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15044605995008 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 186604768256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 % _timestamp‚1.7844866219594004e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 192840565248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538662400 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1017270 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚11809170530816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214286336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14703096555920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1989369 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538662400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214286336 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚827 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 186264816664 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340577792 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚14858359323136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 192965555200 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚14858341804544 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚12002136086016 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2252556800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149414400 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1504382 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚14858359323136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356960818 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚15044964091392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 185924895206 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 186264190464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358096384 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537999223 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358096384 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239511 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 186264190464 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚483 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339921458 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1504384 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚15044964091392 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1504384 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚508771 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚508487 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚14889004411766 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1979633 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 186604768256 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1526043 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚12001986671616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11809146106368 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3493753 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1017272 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 _runtime‚ 98.534315479 _step‚13 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚14889361372584 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 trainer/global_step‚199 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14858341804544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0<ùµ'Ô¯O UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340577792 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚14858359323136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 192965555200 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚14858341804544 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚12002136086016 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2252556800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149414400 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1504382 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚14858359323136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356960818 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚15044964091392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 185924895206 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 186264190464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358096384 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537999223 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358096384 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239511 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 186264190464 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚483 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339921458 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1504384 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚15044964091392 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1504384 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚508771 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚508487 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚14889004411766 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1979633 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 186604768256 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1526043 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚12001986671616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11809146106368 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3493753 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1017272 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 _runtime‚ 98.534315479 _step‚13 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚14889361372584 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 trainer/global_step‚199 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14858341804544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1989369 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 142405120 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1504382 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1979633 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15044605995008 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 186604768256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 % _timestamp‚1.7844866219594004e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 192840565248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538662400 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1017270 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚11809170530816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214286336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14703096555920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1989369 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538662400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214286336 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚827 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 186264816664 ªê9….Õ) _wandbruntime‚98  _runtime‚98‚sªAÖµM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚12061819073536 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14776522638400 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 186853118442 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 187194101760 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1999258 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1989508 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340584960 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 193796634624 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1511886 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214286336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1511886 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚14932551092224 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2252556800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1022351 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12061976869376 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1999258 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 187534686720 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15119727675392 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340584960 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3511144 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1511884 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚11868046863360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356960930 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚511360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157795840 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1533713 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1511884 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133371392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3511144 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14932551092224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537999223 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_!¹³pool.allocated‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3501392 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚15120085778944 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 193930006016 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14963358717482 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538662400 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3501392 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14776505599040 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538662400 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1533421 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚14 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1022353 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 _runtime‚ 98.891795008 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 187534686720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚14932533573632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 187194101760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚199ÁŠ¢]¿&×¹M I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1989508 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340584960 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 193796634624 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1511886 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214286336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1511886 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚14932551092224 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2252556800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1022351 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12061976869376 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1999258 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 187534686720 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15119727675392 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340584960 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3511144 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1511884 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚11868046863360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356960930 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚511360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157795840 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1533713 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1511884 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133371392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3511144 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14932551092224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537999223 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3501392 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚15120085778944 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 193930006016 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14963358717482 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538662400 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3501392 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14776505599040 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538662400 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1533421 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚14 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1022353 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 _runtime‚ 98.891795008 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 187534686720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚14932533573632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 187194101760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚199 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚12061819073536 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14776522638400 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 186853118442 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 187194101760 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1999258²8‚Ù.Ø) _wandbruntime‚98  _runtime‚98‚¼Ùü 2loss/SRqqt2/assignment_loss‚2.4813543346949984 1loss/FBt1/assignment_loss‚0.44751525076017495 /loss/FRt1/detection_loss‚1.6510008874804127 % _timestamp‚1.7844866223176048e+09 _step‚15 /loss/FRt2/assignment_loss‚4.535195262808549 2loss/SRqqt1/assignment_loss‚2.6120472517682285 1loss/SRqqt2/detection_loss‚3.6510051793711527 .loss/FBt1/detection_loss‚32.27286421560815 0loss/FRt1/assignment_loss‚5.3479630129281865 1loss/SRqqt1/detection_loss‚1.2303891462396699 epoch‚0 trainer/global_step‚199 _runtime‚ 98.892302418 1loss/FBt2/assignment_loss‚0.22861077162087895 0loss/FRt2/detection_loss‚0.36426659587174837 .loss/FBt2/detection_loss‚4.641508638681587 &loss/total_loss‚59.463720547833745¤™Ú€ 0loss/FRt1/assignment_loss‚5.3479630129281865 1loss/SRqqt1/detection_loss‚1.2303891462396699 epoch‚0 trainer/global_step‚199 _runtime‚ 98.892302418 1loss/FBt2/assignment_loss‚0.22861077162087895 0loss/FRt2/detection_loss‚0.36426659587174837 .loss/FBt2/detection_loss‚4.641508638681587 &loss/total_loss‚59.463720547833745 2loss/SRqqt2/assignment_loss‚2.4813543346949984 1loss/FBt1/assignment_loss‚0.44751525076017495 /loss/FRt1/detection_loss‚1.6510008874804127 % _timestamp‚1.7844866223176048e+09 _step‚15 /loss/FRt2/assignment_loss‚4.535195262808549 2loss/SRqqt1/assignment_loss‚2.6120472517682285 1loss/SRqqt2/detection_loss‚3.6510051793711527 .loss/FBt1/detection_loss‚32.27286421560815aîôΓÛ‚(:ˆ äµôÒ€‘ûË gpu.0.gpu‚71.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚251.352' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚83.784 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©F /Ü‚(:¤ äµôÒ€¦ˆÌ# proc.memory.rssMB‚ 2778.49609375 cpu‚25.799374543711348 disk.dm-0.out‚24.35693359375 disk.rbd0.in‚0) proc.memory.percent‚16.95859432220459 proc.cpu.threads‚17 network.recv‚64971$ disk./.usageGB‚1174.397117614746 disk.md0.in‚0 disk.md0.out‚ 2.51953125 disk.dm-0.in‚0! disk.dm-1.out‚535.81591796875 disk.rbd0.out‚0' proc.memory.availableMB‚ 115.2421875 network.sent‚231157) disk./.usagePercent‚76.49549699493542 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0& disk.nvme1c1n1.out‚540.05712890625 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚540.05712890625 disk.dm-1.in‚0% disk.nvme0c0n1.out‚25.52001953125# disk.nvme0n1.out‚25.52001953125 disk.nvme1c1n1.in‚0$ memory_percent‚99.29661750793457äö-!Ý‚(:– óµôÒ¾ãË" disk.nvme1n1.out‚ 543.662109375 disk.dm-0.out‚47.12060546875 disk.dm-1.in‚0 cpu‚25.795827175461138 network.sent‚267248 disk.md0.in‚0! disk.dm-1.out‚535.94482421875 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 543.662109375$ memory_percent‚99.29697513580322 proc.memory.rssMB‚2778.5 disk.dm-0.in‚0# disk.nvme0c0n1.out‚ 45.673828125 disk.nvme0n1.in‚0! disk.nvme0n1.out‚ 45.673828125) disk./.usagePercent‚76.49549003766309% disk./.usageGB‚1174.3970108032227 disk.md0.out‚ 2.91796875 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0( proc.memory.availableMB‚ 115.18359375( proc.memory.percent‚16.9586181640625 proc.cpu.threads‚17 network.recv‚69432 disk.nvme1n1.in‚0ì cìžÞ‚(:“ óµôÒçâžÌ gpu.0.gpu‚91.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚257.332' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.77733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00LC0ß+ _wandbruntime‚122  _runtime‚122ô»ö®Äྠ& lr-AdamW/pg1‚6.616662354182313e-05 & lr-AdamW/pg2‚6.616662354182313e-05 trainer/global_step‚249 % _timestamp‚1.7844866456580348e+09 _runtime‚ 122.234427486 _step‚16GèÍÈá & lr-AdamW/pg1‚6.616662354182313e-05 & lr-AdamW/pg2‚6.616662354182313e-05 trainer/global_step‚249 % _timestamp‚1.7844866456580348e+09 _runtime‚ 122.234427486 _step‚164¼äÊ0â+ _wandbruntime‚122  _runtime‚122|Sa²'ã¬O XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340577792 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1907111 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2475235 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18800722271232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚18374383640560 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538662400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚18800722271232 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚14752987158016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚18374400679920 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4364455 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚14993958863360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 _runtime‚ 122.234487486 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚18567930258944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2475235 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1879484 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1879482 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚483 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239511 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 233132590080 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214286336 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 240971705344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1271320 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚635791 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚14993803157504 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131281408 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 trainer/global_step‚249 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 232367960214 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚18801080367616 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214286336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356960818 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚18567930258944 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2484971 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1271322 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚17 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚14752962733568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340577792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 232792012288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358096384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚18606751600774 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚18567947777536 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚636060 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538662400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚18567947777536 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚18801080367616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339921458 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1907382 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚827 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2252556800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 142405120 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 232707881672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155705856 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 240840423936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1879482 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚483 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239511 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 233132590080 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214286336 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 240971705344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1271320 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚635791 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚14993803157504 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131281408 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 trainer/global_step‚249 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 232367960214 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚18801080367616 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214286336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356960818 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 TGDevice¹†2æStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚18567930258944 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2484971 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1271322 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚17 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚14752962733568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340577792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 232792012288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358096384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚18606751600774 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚18567947777536 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚636060 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538662400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚18567947777536 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚18801080367616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339921458 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1907382 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚827 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2252556800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 142405120 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 232707881672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155705856 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 240840423936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18800722271232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚18374383640560 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538662400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚18800722271232 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚14752987158016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚18374400679920 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4364455NŠú0å+ _wandbruntime‚122  _runtime‚122[øÿ9¼&æ¶M bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚14811863490560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2494860 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1886986 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2252556800 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340584960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358103552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2494860 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1276401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133371392 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 241796786176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚14811839066112 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚18681462867420 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214286336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340584960 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1276403 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 233721923584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚18642122028032 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4372094 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2485110 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356960930 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 233636105020 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚18642139546624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844866460154088e+09 _step‚18 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1886984 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚18875843951616 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚638463 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15053635852288 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339921570 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1886986 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚18876202055168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538662400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚18681105906490 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537999223 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1914584 _runtime‚ 122.590564883 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 142405120 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚18447826762400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 241930157568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚18447809723040 trainer/global_step‚249 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15053793648128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538662400 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4381846 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚483 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚18642122028032 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚827+)J÷À&çºM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚18642139546624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844866460154088e+09 _step‚18 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1886984 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚18875843951616 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚638463 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15053635852288 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339921570 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1886986 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚18876202055168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538662400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚18681105906490 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537999223 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1914584 _runtime‚ 122.590564883 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 142405120 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚18447826762400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 241930157568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚18447809723040 trainer/global_step‚249 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15053793648128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538662400 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4381846 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚483 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚18642122028032 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚827 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚14811863490560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2494860 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1886986 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2252556800 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340584960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358103552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2494860 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1276401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133371392 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 241796786176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚14811839066112 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚18681462867420 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214286336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340584960 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1276403 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 233721923584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚18642122028032 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4372094 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2485110 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356960930 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 233636105020 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675Tý<å0è+ _wandbruntime‚122  _runtime‚122Òì³!ÿéù 1loss/SRqqt2/assignment_loss‚2.287386714595638 1loss/FBt1/assignment_loss‚0.27907509699485505 0loss/SRqqt2/detection_loss‚4.515559753734771 &loss/total_loss‚45.102758890166115 trainer/global_step‚249 _runtime‚ 122.591049393 _step‚19 /loss/FRt2/assignment_loss‚4.539994339424919 .loss/FBt2/detection_loss‚7.462650904322465 /loss/FRt1/assignment_loss‚5.314382993257963 0loss/FBt2/assignment_loss‚0.1913051969974979 0loss/FRt2/detection_loss‚0.41169441940529145 0loss/SRqqt1/detection_loss‚0.951876676488513 % _timestamp‚1.7844866460163546e+09 2loss/SRqqt1/assignment_loss‚2.2816811173917126 /loss/FRt1/detection_loss‚1.6214061992354372 /loss/FBt1/detection_loss‚15.245745478317055 epoch‚0O9iêý /loss/FRt1/assignment_loss‚5.314382993257963 0loss/FBt2/assignment_loss‚0.1913051969974979 0loss/FRt2/detection_loss‚0.41169441940529145 0loss/SRqqt1/detection_loss‚0.951876676488513 % _timestamp‚1.7844866460163546e+09 2loss/SRqqt1/assignment_loss‚2.2816811173917126 /loss/FRt1/detection_loss‚1.6214061992354372 /loss/FBt1/detection_loss‚15.245745478317055 epoch‚0 1loss/SRqqt2/assignment_loss‚2.287386714595638 1loss/FBt1/assignment_loss‚0.27907509699485505 0loss/SRqqt2/detection_loss‚4.515559753734771 &loss/total_loss‚45.102758890166115 trainer/global_step‚249 _runtime‚ 122.591049393 _step‚19 /loss/FRt2/assignment_loss‚4.539994339424919 .loss/FBt2/detection_loss‚7.4626509043224657Ûííë‚(:’ ‚¶ôÒ½·½Ë% disk./.usageGB‚1174.3969955444336 disk.dm-0.out‚47.69091796875 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0# disk.nvme0c0n1.out‚ 46.572265625 disk.nvme0n1.in‚0! disk.nvme0n1.out‚ 46.572265625 disk.nvme1c1n1.in‚0 network.sent‚270698 disk.md0.out‚ 3.2734375 disk.dm-1.out‚ 536.166015625& disk.nvme1c1n1.out‚544.31298828125 disk.nvme1n1.in‚0 proc.memory.rssMB‚2778.5( proc.memory.percent‚16.9586181640625 cpu‚25.78964026221494 network.recv‚73498) disk./.usagePercent‚76.49548904376705 disk.md0.in‚0$ disk.nvme1n1.out‚544.31298828125$ proc.memory.availableMB‚114.4375 disk.dm-0.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚99.30152893066406 proc.cpu.threads‚17–qç“ì‚(:ˆ ‚¶ôÒ¤àñË gpu.0.gpu‚91.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚52.0 gpu.0.powerWatts‚117.897' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.299 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_ùóô0í+ _wandbruntime‚145  _runtime‚145}±¢žÄî¾ & lr-AdamW/pg1‚7.945309413255067e-05 & lr-AdamW/pg2‚7.945309413255067e-05 trainer/global_step‚299 % _timestamp‚1.7844866693271341e+09 _runtime‚ 145.902104512 _step‚20…0Èï _runtime‚ 145.902104512 _step‚20 & lr-AdamW/pg1‚7.945309413255067e-05 & lr-AdamW/pg2‚7.945309413255067e-05 trainer/global_step‚299 % _timestamp‚1.7844866693271341e+09P'?0ð+ _wandbruntime‚145  _runtime‚145bѯM> ñ¬O [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22277536231936 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 279319834112 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚17985577828352 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2254582 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22556838547456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚483 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 279660411904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2970837 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356960818 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239511 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 _runtime‚ 145.903571804 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538662400 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358096384 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚827 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1525370 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚17696803785216 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 ó¶'ò°O \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22277536231936 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 288938137600 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5225419 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22277518713344 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2970837 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22277518713344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚22045704803920 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 279150946680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 279660411904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 trainer/global_step‚299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 279660411904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2970837 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356960818 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239511 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 _runtime‚ 145.903571804 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538662400 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358096384 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚827 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1525370 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚17696803785216 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22277536231936 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 279319834112 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚17985577828352 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2254582 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22556838547456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚483 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22398853095498 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22632318331392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2295565 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚827 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22632318331392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 280249745408 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161990144 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 289753533440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22119113847040 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5252548 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚17755655693312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1530451 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5242796 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚17755680117760 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5242796 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22351728001024 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358103552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2252556800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1530453 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2262084 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2980712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚22351710482432 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 279739248458 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340584960 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚765112 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 289891099136 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5252548 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2990462 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7844866696833763e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2295275 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 280079170028 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934ÑMnAÇõºM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚18045409226752 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 137565696 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 280590330368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2980712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚483 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 _runtime‚ 146.258582597 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22398853095498 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22632318331392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2295565 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚827 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22632318331392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 280249745408 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161990144 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 289753533440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22119113847040 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5252548 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚17755655693312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1530451 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 F:DeviceStatsMonitor.on_train_bat3¿/”ùch_end/allocation.all.freed‚5242796 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚17755680117760 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5242796 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22351728001024 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358103552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2252556800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1530453 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2262084 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2980712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚22351710482432 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 279739248458 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340584960 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚765112 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 289891099136 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5252548 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2990462 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7844866696833763e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2295275 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 280079170028 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26312954823680 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 325847655936 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3466439 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214286336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538662400 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚26313312920064 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 326188233728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 142405120 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3466439 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2629682 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚484 trainer/global_step‚349 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538662400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚889390 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚25 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6096121 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚25987107167744 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 325847655936 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 336755050496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚20977506744320 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239511 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚25987124686336 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚25987107167744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚25717008927920 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2629684 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340577792 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚26042602939608 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚26042245978790 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2669092 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2252556800 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131281408 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214286336 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚20640620412416 _runtime‚ 169.738145527 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚828 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358096384 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6105859 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚20977351038464 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 326188233728 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358096384 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155705856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0l†×µ'‚¯O L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6105859 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚20977351038464 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 326188233728 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358096384 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155705856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537999223 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 325254090230 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1779420 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚26313312920064 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1779422 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚25716991888560 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚25987124686336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356960818 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 $ _timestamp‚1.784486693163292e+09 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3476175 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26312954823680 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 325847655936 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3466439 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214286336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538662400 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚26313312920064 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 326188233728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 142405120 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3466439 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2629682 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚484 trainer/global_step‚349 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538662400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚889390 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚25 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6096121 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚25987107167744 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 325847655936 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 336755050496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚20977506744320 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239511 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚25987124686336 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚25987107167744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚25717008927920 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2629684 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340577792 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚26042602939608 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚26042245978790 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2669092 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2252556800 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131281408 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214286336 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚20640620412416 _runtime‚ 169.738145527 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚828 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358096384;™ÎÖ0ƒ+ _wandbruntime‚170  _runtime‚170Ö8¼nb „¶M ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚20699472320512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 142405120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 326522235036 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚26388434607616 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 337842369536 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2611660028Ðç°Z4506 trainer/global_step‚349 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21037189707264 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚26061316455424 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 326777567232 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6113498 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6123250 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124982784 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6123250 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚891851 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2637184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 326182313466 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1784501 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2676352 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3486064 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚892130 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚25790417971040 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 327118152192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21037339114496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚ 170.095564741 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6113498 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358103552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚26061316455424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340584960 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 327118152192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚828 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚261169572454364!nTÀ&…ºM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚26061298936832 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2637186 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 326182313466 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1784501 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2676352 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3486064 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚892130 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚25790417971040 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 327118152192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21037339114496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚ 170.095564741 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6113498 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358103552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚26061316455424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340584960 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 327118152192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚828 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚26116957245436 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚20699472320512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 142405120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 326522235036 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚26388434607616 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 337842369536 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚26116600284506 trainer/global_step‚349 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21037189707264 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚26061316455424 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 326777567232 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6113498 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6123250 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124982784 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6123250 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚891851 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2637184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 193.582407545 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30069071099904 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 372037076696 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3962041 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1015620 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚30069071099904 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538662400 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 372716055552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚29388313051920 _step‚29 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚23584437039616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1015917 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 trainer/global_step‚399 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3049389 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3962041 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23584412615168 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 384747295744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 372375477760 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3004782 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29760350128616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3971777 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30069429196288 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚29696695622144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚29696713140736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2252556800 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6966823 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2033470 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚23969159910912 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214286336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29388296012560 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 372716055552 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358096384 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 371697155238 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538662400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚23969324005376 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29696713140736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358096384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚828 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2033472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚484 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3004784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3971777 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚29759993167798 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537999223 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚29696695622144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340577792 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368Ê6Õ“þ²O ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139670016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 142405120 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339921458 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3004782 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6976561 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚297 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356960818 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239511 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3004784 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3049090 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 164094464 % _timestamp‚1.7844867170041974e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 193.582407545 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30069071099904 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 372037076696 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3962041 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_po™Á\¶ºol.freed‚1015620 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚30069071099904 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538662400 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 372716055552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚29388313051920 _step‚29 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚23584437039616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1015917 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 trainer/global_step‚399 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3049389 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3962041 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23584412615168 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 384747295744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 372375477760 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3004782 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29760350128616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3971777 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30069429196288 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚29696695622144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚29696713140736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2252556800 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6966823 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2033470 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚23969159910912 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214286336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29388296012560 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 372716055552 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358096384 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 371697155238 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538662400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚23969324005376 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29696713140736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358096384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚828 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2033472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚484 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3004784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3971777 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚29759993167798 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537999223 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚29696695622144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340577792 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚30069429196288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚30144550883840 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 385840510976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23643313372160 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚29770904909824 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6984200 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340584960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3056956 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 372625378474 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139662848 _runtime‚ 193.936911634 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚29834347473514 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 373645974016 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3981666 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3056658 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2252556800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214286336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚24029153883136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2038553 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6993952 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 $ _timestamp‚1.784486717361757e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 373645974016 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚29770904909824 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚30 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358103552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6993952 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 164087296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3971916 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚23643288947712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29461722095040ZTºaÁ&“»M EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3056658 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2252556800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214286336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚24029153883136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2038553 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6993952 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 $ _timestamp‚1.784486717361757e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 373645974016 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚29770904909824 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚30 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358103552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6993952 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 164087296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3971916 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚23643288947712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29461722095040 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚29770887391232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚30144550883840 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 385840510976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23643313372160 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚29770904909824 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6984200 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340584960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3056956 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 372625378474 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139662848 _runtime‚ 193.936911634 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚29834347473514 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 373645974016 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3981666 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239511 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚33406301595136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214286336 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 432868120576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358096384 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1141646 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚33477740356806 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4457643 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7847263 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2287522 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚33478097317624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1141377 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 142405120 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33825187376128 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2287520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚484 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26528229242368 _step‚33 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3379882 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚26961121787392 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159900160 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚01ÏÏu¸' ²O XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339921458 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚33825545472512 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3429168 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537999223 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚33406284076544 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 419243877376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538662400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 432732644864 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚7837525 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 418903299584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340577792 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 418903299584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239511 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚33406301595136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214286336 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 432868120576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358096384 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1141646 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚33477740356806 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4457643 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7847263 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2287522 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚33478097317624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1141377 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 142405120 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33825187376128 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2287520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚484 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26528229242368 _step‚33 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3379882 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚26961121787392 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159900160 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚33825187376128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚33059600136560 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214286336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538662400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26528253666816 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358096384 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 419243877376 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3428897 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7844867412412543e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚33406301595136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚33059617175920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 _runtime‚ 217.816326882 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3379884 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 418480141704 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3379882 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1!…N,ê0¡+ _wandbruntime‚218  _runtime‚218 \˜%¾&¢¸M QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4477268 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1143890 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚33480475845632 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4467518 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 433684386816 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 419833210880 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚484 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚33552094662522 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 419408365052 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537999223 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3436795 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356960930 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3387384 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26587129999360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 142405120 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33900667160064 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _step‚34 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2292601 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 218.194240478 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7864654 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2292603 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3387386 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161990144 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 137565696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 trainer/global_step‚449 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358103552 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚7854902 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3387384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚304 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚27020789961728 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27020951951872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7854902 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340584960 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 419833210880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3436491 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚33133026219040 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4467518 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚33552451623452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚828 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚33480493364224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 % _timestamp‚1.7844867416191034e+09 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26587105574912 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚33133043258400.AW £¼M J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3436491 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚33133026219040 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4467518 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚33552451623452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚828 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚33480493364224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 % _timestamp‚1.7844867416191034e+09 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26587105574912 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚33133043258400 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4477268 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1143890 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368EÓ£ A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚33480475845632 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4467518 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 433684386816 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 419833210880 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚484 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚33552094662522 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 419408365052 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537999223 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3436795 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356960930 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3387384 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26587129999360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 142405120 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33900667160064 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _step‚34 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2292601 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 218.194240478 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7864654 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2292603 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3387386 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161990144 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 137565696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 trainer/global_step‚449 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358103552 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚7854902 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3387384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚304 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚27020789961728 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27020951951872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7854902 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340584960 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 419833210880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446"5nÀý0¤+ _wandbruntime‚218  _runtime‚218Ã3ëþ¥ø _step‚35 1loss/SRqqt1/detection_loss‚1.1201771094548358 trainer/global_step‚449 /loss/FRt2/assignment_loss‚4.516511014950129 1loss/FBt1/assignment_loss‚0.15800365628789828 .loss/FRt1/detection_loss‚1.330981749802681 .loss/FBt1/detection_loss‚8.261157419483013 .loss/FBt2/detection_loss‚6.285076532078647 _runtime‚ 218.194982959 /loss/FRt1/assignment_loss‚4.072964031708064 1loss/FBt2/assignment_loss‚0.26894461801660535 0loss/FRt2/detection_loss‚0.33543554752642746 $loss/total_loss‚33.1040882942071 epoch‚0 2loss/SRqqt1/assignment_loss‚1.5447784554479378 2loss/SRqqt2/assignment_loss‚1.8094329884526326 0loss/SRqqt2/detection_loss‚3.400625170998232 % _timestamp‚1.7844867416201723e+09±$¦ü 1loss/SRqqt1/detection_loss‚1.1201771094548358 trainer/global_step‚449 /loss/FRt2/assignment_loss‚4.516511014950129 1loss/FBt1/assignment_loss‚0.15800365628789828 .loss/FRt1/detection_loss‚1.330981749802681 .loss/FBt1/detection_loss‚8.261157419483013 .loss/FBt2/detection_loss‚6.285076532078647 _runtime‚ 218.194982959 /loss/FRt1/assignment_loss‚4.072964031708064 1loss/FBt2/assignment_loss‚0.26894461801660535 0loss/FRt2/detection_loss‚0.33543554752642746 $loss/total_loss‚33.1040882942071 epoch‚0 2loss/SRqqt1/assignment_loss‚1.5447784554479378 2loss/SRqqt2/assignment_loss‚1.8094329884526326 0loss/SRqqt2/detection_loss‚3.400625170998232 % _timestamp‚1.7844867416201723e+09 _step‚35#™%~·+§‚(:  ܶôÒƒÑãË network.recv‚100025 disk.md0.in‚0 disk.dm-0.in‚0 disk.dm-1.in‚0! disk.dm-1.out‚537.55908203125& disk.nvme0c0n1.out‚394.60400390625( proc.memory.availableMB‚ 112.69921875) disk./.usagePercent‚76.52569031447507 disk.nvme0n1.in‚0$ disk.nvme1c1n1.out‚ 738.677734375 disk.rbd0.out‚0! proc.memory.rssMB‚ 2778.515625) proc.memory.percent‚16.95871353149414 proc.cpu.threads‚17% disk./.usageGB‚1174.8606605529785 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0$ memory_percent‚99.31213855743408 cpu‚25.826960644687794$ disk.nvme0n1.out‚394.60400390625" disk.nvme1n1.out‚ 738.677734375 disk.md0.out‚7.90625 disk.dm-0.out‚579.0322265625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0 network.sent‚469874eÜþkž¨‚(:“ ܶôÒš§ÚÌ gpu.0.gpu‚90.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚212.048' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚70.68266666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã£Ñ!©‚(:– ë¶ôÒ¢…ÓË network.recv‚104834" disk.nvme1c1n1.out‚ 739.1640625 disk.nvme1n1.out‚ 739.1640625 disk.dm-1.in‚0 network.sent‚519048% disk./.usageGB‚1174.8611183166504" disk.nvme0c0n1.out‚ 396.1328125 disk.nvme0n1.out‚ 396.1328125 disk.dm-0.in‚0! disk.dm-0.out‚580.32470703125! disk.dm-1.out‚537.59033203125$ memory_percent‚99.31387901306152) disk./.usagePercent‚76.52572013135648 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0' proc.memory.availableMB‚ 112.4140625! proc.memory.rssMB‚ 2778.515625) proc.memory.percent‚16.95871353149414 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 8.22265625 cpu‚25.88250304601845–MNbžª‚(:“ ë¶ôÒ¦ä‡Ì gpu.0.gpu‚60.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚248.051' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚82.68366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i,®0«+ _wandbruntime‚243  _runtime‚243í}R¢Å¬¿ ' lr-AdamW/pg2‚0.00013259897649546082 trainer/global_step‚499 $ _timestamp‚1.784486766777938e+09 _runtime‚ 243.352919744 _step‚36 ' lr-AdamW/pg1‚0.00013259897649546082t¯ÄÉ­Ã ' lr-AdamW/pg2‚0.00013259897649546082 trainer/global_step‚499 $ _timestamp‚1.784486766777938e+09 _runtime‚ 243.352919744 _step‚36 ' lr-AdamW/pg1‚0.00013259897649546082$«IÀ¹0®+ _wandbruntime‚243  _runtime‚243ïY–´'¯®O WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 trainer/global_step‚499 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358096384 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 465431121408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚29952763153920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3809900 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37581661748736 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚37115890049536 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 464583285254 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538662400 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚29952920956928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 142405120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚828 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚37115872530944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37581303652352 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 465431121408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚29472070294016 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7844867667789805e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚37115872530944 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8717965 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3754984 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 480717284352 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚36730904260560 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3754982 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340577792 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3809605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚484 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚37195487545814 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4962981 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339921458 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340577792 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37581661748736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚36730921299920 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214286336 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1268328 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356960818 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239511 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538662400 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8708227 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚37195844506632 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37581303652352 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4953245 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚29472045869568 _step‚37 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚37115890049536 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 465771699200 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3754982 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157803008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3754984 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4953245N³*¸'°²O ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133378560 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 464923206712 _runtime‚ 243.353982556 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4962981 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358096384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2541570 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8708227 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚37195844506632 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37581303652352 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4953245 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚29472045869568 _step‚37 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚37115890049536 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 465771699200 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3754982 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157803008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3754984 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4953245 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 trainer/global_step‚499 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358096384 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 465431121408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚29952763153920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3809900 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37581661748736 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚37115890049536 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 464583285254 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538662400 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚29952920956928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 142405120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚828 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚37115872530944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37581303652352 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 465431121408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚29472070294016 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7844867667789805e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚37115872530944 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8717965 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3754984 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 480717284352 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚36730904260560 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3754982 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340577792 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3809605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚484 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚37195487545814 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4962981 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339921458 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340577792 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37581661748736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚36730921299920 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214286336 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1268328 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356960818 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239511 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538662400 HDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8735356 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3817479 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚29530922202112 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537999223 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 142405120 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚37269841851530 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358103552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚37190064300032 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3762486 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3817187 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340584960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 trainer/global_step‚499 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340584960 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37656425332736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8725604 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 465511508490 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 466701617664 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37656783436288 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37656783436288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4972870 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1270826 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚37270198812460 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538662400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159892992 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 481807529984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3762486 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4963120 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚37190081818624 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 466361032704 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 466701617664 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37656783436288 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37656783436288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4972870 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1270826 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚37270198812460 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538662400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159892992 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 481807529984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3762486 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4963120 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚37190081818624 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 466361032704 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8735356 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3817479 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚29530922202112 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537999223 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 142405120 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚37269841851530 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358103552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚37190064300032 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3762486 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3817187 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340584960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 trainer/global_step‚499 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340584960 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37656425332736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0&Þ¡=·0´+ _wandbruntime‚243  _runtime‚243w,‡åµû 2loss/SRqqt1/assignment_loss‚1.4787533394020536 0loss/FRt2/detection_loss‚0.33465568144005287 0loss/SRqqt2/detection_loss‚3.248318056940858 /loss/FRt1/assignment_loss‚3.942218112194632 2loss/SRqqt2/assignment_loss‚1.7342916981518617 /loss/FBt2/detection_loss‚14.053071953362005 _runtime‚ 243.727578883 /loss/FRt2/assignment_loss‚4.482508475963886 1loss/FBt1/assignment_loss‚0.14699303155997162 1loss/SRqqt1/detection_loss‚0.9673677741650296 epoch‚0 trainer/global_step‚499 % _timestamp‚1.7844867671528745e+09 0loss/FBt2/assignment_loss‚0.6353334087584896 /loss/FRt1/detection_loss‚1.3698413105121767 .loss/FBt1/detection_loss‚5.582283917684374 &loss/total_loss‚37.975636760135394 _step‚39˲Üá¶ÿ /loss/FRt1/assignment_loss‚3.942218112194632 2loss/SRqqt2/assignment_loss‚1.7342916981518617 /loss/FBt2/detection_loss‚14.053071953362005 _runtime‚ 243.727578883 /loss/FRt2/assignment_loss‚4.482508475963886 1loss/FBt1/assignment_loss‚0.14699303155997162 1loss/SRqqt1/detection_loss‚0.9673677741650296 epoch‚0 trainer/global_step‚499 % _timestamp‚1.7844867671528745e+09 0loss/FBt2/assignment_loss‚0.6353334087584896 /loss/FRt1/detection_loss‚1.3698413105121767 .loss/FBt1/detection_loss‚5.582283917684374 &loss/total_loss‚37.975636760135394 _step‚39 2loss/SRqqt1/assignment_loss‚1.4787533394020536 0loss/FRt2/detection_loss‚0.33465568144005287 0loss/SRqqt2/detection_loss‚3.248318056940858'âïý(,·‚(:¡ ú¶ôÒÙßË disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚739.8935546875 disk.md0.in‚0 disk.md0.out‚ 8.62109375! disk.dm-0.out‚581.12939453125! proc.memory.rssMB‚ 2778.515625 disk.nvme0c0n1.in‚0 disk.nvme1n1.in‚0 disk.dm-1.out‚ 537.736328125" memory_percent‚99.314284324646( proc.memory.availableMB‚ 112.34765625) proc.memory.percent‚16.95871353149414 cpu‚25.91320910006449 network.sent‚535512 network.recv‚109072) disk./.usagePercent‚76.52545724585204% disk./.usageGB‚1174.8570823669434 disk.rbd0.out‚0$ disk.nvme0n1.out‚397.20068359375 disk.rbd0.in‚0& disk.nvme0c0n1.out‚397.20068359375 disk.nvme0n1.in‚0# disk.nvme1n1.out‚739.8935546875 disk.dm-0.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17~<.Úž¸‚(:“ ú¶ôÒ€³”Ì gpu.0.gpu‚98.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚253.594' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.53133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 0Ã80¹+ _wandbruntime‚269  _runtime‚269W3äÊź¿ ' lr-AdamW/pg1‚0.00014588544708618836 ' lr-AdamW/pg2‚0.00014588544708618836 trainer/global_step‚549 $ _timestamp‚1.784486792663012e+09 _runtime‚ 269.237969836 _step‚40? Oêɻà ' lr-AdamW/pg1‚0.00014588544708618836 ' lr-AdamW/pg2‚0.00014588544708618836 trainer/global_step‚549 $ _timestamp‚1.784486792663012e+09 _runtime‚ 269.237969836 _step‚40(//0¼+ _wandbruntime‚269  _runtime‚269ª›ÿ³'½­O M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214286336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚40825460985344 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 $ _timestamp‚1.784486792664222e+09 _runtime‚ 269.239152289 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41337778024960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚828 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚32944733143552 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 528714940928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1395243 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 528846222336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239511 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214286336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚484 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9578929 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356960818 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 511366271720 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339921458 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2795622 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340577792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚32415886921216 _step‚41 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 512299521024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538662400 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚41337778024960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚40402208384560 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32944577437696 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 511958943232 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538662400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 512299521024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 142405120 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537999223 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 trainer/global_step‚549 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4130084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340577792 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚40913591695640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚40825478503936 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4190577 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚32415862496768 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41337419928576 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚40825460985344 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 511958943232 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5458583 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41337419928576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131281408 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358096384 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4130084 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5448847 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2795620 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚40402225423920 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9588667 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1394957 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5458583 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358096384 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5448847 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 511026350262 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259ç§Ñ!1¾±O =7DeviceStatsMonitor.on_train_batch_startó 1Þ†'/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 511366271720 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339921458 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2795622 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340577792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚32415886921216 _step‚41 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 512299521024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538662400 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚41337778024960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚40402208384560 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32944577437696 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 511958943232 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538662400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 512299521024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 142405120 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537999223 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 trainer/global_step‚549 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4130084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340577792 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚40913591695640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚40825478503936 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4190577 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚32415862496768 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41337419928576 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚40825460985344 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 511958943232 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5458583 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41337419928576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131281408 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358096384 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4130084 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5448847 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2795620 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚40402225423920 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9588667 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1394957 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5458583 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358096384 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5448847 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 511026350262 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214286336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚40825460985344 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 $ _timestamp‚1.784486792664222e+09 _runtime‚ 269.239152289 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41337778024960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚828 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚32944733143552 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 528714940928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1395243 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 528846222336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239511 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214286336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚484 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9578929 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356960818)¿ ¤”0¿+ _wandbruntime‚269  _runtime‚269£At½&À·M VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340584960 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4198463 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚40475651506400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4137584 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚40899670273024 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚33004572981760 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 511954573498 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2800701 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 529809728000 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4137586 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9606058 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 512294495068 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚41412899712512 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚41412899712512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1397470 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚40987589040538 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9596306 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚41412541608960 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 137565696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340584960 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚40899652754432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5458722 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 142405120 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4137586 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1397760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5468472 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 269.61716511 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538662400 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538662400 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4198171 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9606058 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214286336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537999223 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358103552 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4137584 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚32474763253760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 trainer/global_step‚549ýM´Á&Á»M WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358103552 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4137584 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚32474763253760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 trainer/global_step‚549 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340584960 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4198463 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚40475651506400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4137584 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚40899670273024 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚33004572981760 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 511954573498 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2800701 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 529809728000 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4137586 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9606058 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 512294495068 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚41412899712512 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚41412899712512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1397470 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚40987589040538 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9596306 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚41412541608960 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 137565696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340584960 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚40899652754432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5458722 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 142405120 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4137586 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1397760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5468472 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 269.61716511 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538662400 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538662400 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4198171 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9606058 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214286336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537999223*Î%š™0Â+ _wandbruntime‚269  _runtime‚2694ÍËøÃû -loss/FRt2/assignment_loss‚4.0701031037645 1loss/FBt1/assignment_loss‚0.14538740972802514 1loss/FBt2/assignment_loss‚0.42649991329138476 1loss/SRqqt2/detection_loss‚2.6330843298027338 epoch‚0 _step‚43 2loss/SRqqt1/assignment_loss‚1.5025962939333615 /loss/FRt1/detection_loss‚1.3491593042810759 /loss/FRt2/detection_loss‚0.4066795631346662 0loss/SRqqt1/detection_loss‚1.159431933764348 /loss/FBt2/detection_loss‚11.657613701054029 % _timestamp‚1.7844867930434673e+09 2loss/SRqqt2/assignment_loss‚1.7491772627983337 /loss/FBt1/detection_loss‚4.8552647922582315 _runtime‚ 269.618392583 0loss/FRt1/assignment_loss‚3.6080122776031494 &loss/total_loss‚33.563009885413834 trainer/global_step‚549ž³? Äÿ /loss/FRt1/detection_loss‚1.3491593042810759 /loss/FRt2/detection_loss‚0.4066795631346662 0loss/SRqqt1/detection_loss‚1.159431933764348 /loss/FBt2/detection_loss‚11.657613701054029 % _timestamp‚1.7844867930434673e+09 2loss/SRqqt2/assignment_loss‚1.7491772627983337 /loss/FBt1/detection_loss‚4.8552647922582315 _runtime‚ 269.618392583 0loss/FRt1/assignment_loss‚3.6080122776031494 &loss/total_loss‚33.563009885413834 trainer/global_step‚549 -loss/FRt2/assignment_loss‚4.0701031037645 1loss/FBt1/assignment_loss‚0.14538740972802514 1loss/FBt2/assignment_loss‚0.42649991329138476 1loss/SRqqt2/detection_loss‚2.6330843298027338 epoch‚0 _step‚43 2loss/SRqqt1/assignment_loss‚1.5025962939333615+‹ö¹ª-Å‚(:¢ ‰·ôÒºŽîË proc.cpu.threads‚17 network.recv‚113353% disk./.usageGB‚1174.9197578430176& disk.nvme1c1n1.out‚740.39306640625 disk.nvme1n1.in‚0 disk.dm-0.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 112.18359375) proc.memory.percent‚16.95871353149414# disk.nvme0n1.out‚404.5673828125 disk.md0.in‚0 cpu‚25.957398820694866 disk.nvme0n1.in‚0$ disk.nvme1n1.out‚740.39306640625 disk.md0.out‚8.796875! disk.dm-0.out‚588.55908203125 disk.dm-1.in‚0 disk.rbd0.out‚0$ memory_percent‚99.31528568267822! proc.memory.rssMB‚ 2778.515625 network.sent‚569563) disk./.usagePercent‚76.52953967386533 disk.nvme0c0n1.in‚0% disk.nvme0c0n1.out‚404.5673828125 disk.nvme1c1n1.in‚0 disk.dm-1.out‚ 537.783203125G+#Y*Æ‚(:“ ‰·ôÒ£Û´Ì gpu.0.gpu‚51.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚54.0 gpu.0.powerWatts‚191.205' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚63.73500000000001 gpu.µn…t0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04º+Ç‚(:  ˜·ôÒ´ŠÄË) proc.memory.percent‚16.95871353149414 cpu‚25.99639735318069 disk.md0.in‚0) disk./.usagePercent‚76.52953445591109% disk.nvme0c0n1.out‚405.6572265625 disk.dm-1.in‚0 disk.dm-1.out‚ 538.01171875 proc.cpu.threads‚17$ disk./.usageGB‚1174.919677734375! disk.dm-0.out‚589.23486328125 disk.rbd0.in‚0! proc.memory.rssMB‚ 2778.515625 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚741.23876953125 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚741.23876953125$ memory_percent‚99.31559562683105' proc.memory.availableMB‚ 112.1328125 network.sent‚616676 network.recv‚118360# disk.nvme0n1.out‚405.6572265625 disk.md0.out‚ 9.27734375 disk.dm-0.in‚0 disk.rbd0.out‚0õ¾QžÈ‚(:“ ˜·ôÒãÿýË gpu.0.gpu‚51.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚54.0 gpu.0.powerWatts‚259.617' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.53900000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i_CÍ0É+ _wandbruntime‚295  _runtime‚295·µ%àÄʾ _runtime‚ 295.224962616 _step‚44 & lr-AdamW/pg1‚0.0001591719176769159 & lr-AdamW/pg2‚0.0001591719176769159 trainer/global_step‚599 % _timestamp‚1.7844868186499746e+09#WâÈË & lr-AdamW/pg1‚0.0001591719176769159 & lr-AdamW/pg2‚0.0001591719176769159 trainer/global_step‚599 % _timestamp‚1.7844868186499746e+09 _runtime‚ 295.224962616 _step‚44,Ù¯Ú0Ì+ _wandbruntime‚295  _runtime‚295'WAÕ¸'ͲO bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 576843099136 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚45093894301184 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚44535049439744 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5944449 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35359679123968 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5954185 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339921458 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5944449 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 558827342848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356960818 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537999223 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚45093536204800 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚35936546647552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚44073529547920 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340577792 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1522778 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 557469415270 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4505184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214286336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340577792 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 576711817728 % _timestamp‚1.7844868186510217e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚10449631 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 558827342848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚44073512508560 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4505182 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4505182 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4572448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 _runtime‚ 295.225980128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚828 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3049670 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35359703548416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44631338884648 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4572738 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358096384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚484 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131281408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 trainer/global_step‚599 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 558486765056 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 558486765056 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155705856 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚35936390941696 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214286336 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4505184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45093536204800 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚44535066958336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3049672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 _step‚45 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1523066 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239511 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 557809336728 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358096384 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚44535066958336 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5954185 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 144502272 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44630981923830 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚45093894301184îzÄà¼'ζO OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5944449 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35359679123968 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5954185 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339921458 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5944449 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 558827342848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356960818 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537999223 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚45093536204800 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚35936546647552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚44073529547920 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340577792 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1522778 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 557469415270 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4505184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214286336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340577792 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 576711817728 % _timestamp‚1.7844868186510217e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚10449631 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 558827342848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚44073512508560 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4505182 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4505182 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4572448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 _runtime‚ 295.225980128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚828 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3049670 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35359703548416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44631338884648 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4572738 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358096384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚484 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131281408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 trainer/global_step‚599 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 558486765056 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 558486765056 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155705856 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚35936390941696 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214286336 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4505184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45093536204800 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚44535066958336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3049672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 _step‚45 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1523066 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239511 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 557809336728 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358096384 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚44535066958336 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5954185 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 144502272 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44630981923830 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚45093894301184 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 576843099136 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚45093894301184 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚44535049439744-vb$a0Ï+ _wandbruntime‚295  _runtime‚295ÊÁÀ&кM MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4512684 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚484 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45169015988736 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44609241208832 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10476760 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340584960 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339921570 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5954324 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3054751 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5964074 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚35996227661824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 144502272 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35418579880960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35418555456512 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4512684 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚44146955630400 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 559416676352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1525276 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚44609241208832 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3054753 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5954324 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 559416676352 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4580027 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340584960 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5964074 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚10476760 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1525568 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155698688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131274240 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 # _timestamp‚1.78448681902629e+09 _step‚46 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214286336 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚45168657885184 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4512686 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358103552 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 558397638506 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10467008 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 577672205312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚44146938591040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚44609258727424 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 E8DeviceStatsMonitor.on_train_batch_enfš ¥$d/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5954324 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3054751 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5964074 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚35996227661824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 144502272 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35418579880960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35418555456512 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4512684 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚44146955630400 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 559416676352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1525276 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚44609241208832 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3054753 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5954324 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 559416676352 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4580027 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340584960 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5964074 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚10476760 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1525568 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155698688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131274240 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 # _timestamp‚1.78448681902629e+09 _step‚46 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214286336 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚45168657885184 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4512686 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358103552 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 558397638506 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10467008 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 577672205312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚44146938591040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚44609258727424 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45169015988736 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44609241208832 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10476760 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340584960 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339921570.y¼Çª0Ò+ _wandbruntime‚295  _runtime‚295í/ô·Óû 0loss/FRt2/detection_loss‚0.35177107423105075 1loss/SRqqt2/detection_loss‚3.2530403405181034 % _timestamp‚1.7844868190273838e+09 _runtime‚ 295.602115189 /loss/FRt1/assignment_loss‚3.177783480006687 0loss/FBt2/assignment_loss‚0.4119705995273342 _step‚47 /loss/FRt2/assignment_loss‚4.152469938052328 2loss/SRqqt2/assignment_loss‚1.4299233287572861 0loss/FBt1/assignment_loss‚0.2060714780305346 /loss/FRt1/detection_loss‚1.3807469584595782 .loss/FBt1/detection_loss‚4.113373361549647 /loss/FBt2/detection_loss‚10.432943079334068 &loss/total_loss‚31.204776870622293 trainer/global_step‚599 2loss/SRqqt1/assignment_loss‚1.4145143829397302 1loss/SRqqt1/detection_loss‚0.8801688492159442 epoch‚0«4ãŠÔÿ /loss/FRt1/assignment_loss‚3.177783480006687 0loss/FBt2/assignment_loss‚0.4119705995273342 _step‚47 /loss/FRt2/assignment_loss‚4.152469938052328 2loss/SRqqt2/assignment_loss‚1.4299233287572861 0loss/FBt1/assignment_loss‚0.2060714780305346 /loss/FRt1/detection_loss‚1.3807469584595782 .loss/FBt1/detection_loss‚4.113373361549647 /loss/FBt2/detection_loss‚10.432943079334068 &loss/total_loss‚31.204776870622293 trainer/global_step‚599 2loss/SRqqt1/assignment_loss‚1.4145143829397302 1loss/SRqqt1/detection_loss‚0.8801688492159442 epoch‚0 0loss/FRt2/detection_loss‚0.35177107423105075 1loss/SRqqt2/detection_loss‚3.2530403405181034 % _timestamp‚1.7844868190273838e+09 _runtime‚ 295.602115189/Ö¥o+Õ‚(:  §·ôÒÑŠ€Ì disk.nvme1n1.in‚0! disk.dm-0.out‚593.00439453125 disk.dm-1.in‚0 disk.rbd0.out‚0# memory_percent‚99.3175745010376) proc.memory.percent‚16.95871353149414 network.recv‚122066 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚743.62158203125$ disk.nvme1n1.out‚743.62158203125 disk.dm-0.in‚0 disk.dm-1.out‚ 538.0546875 disk.rbd0.in‚0! proc.memory.rssMB‚ 2778.515625% disk./.usageGB‚1172.0208740234375 disk.nvme0c0n1.in‚0 disk.md0.out‚ 9.4921875( proc.memory.availableMB‚ 111.80859375 proc.cpu.threads‚17) disk./.usagePercent‚76.34071806047467% disk.nvme0c0n1.out‚407.5634765625 disk.nvme0n1.in‚0 disk.md0.in‚0 cpu‚26.031529839921685 network.sent‚635905# disk.nvme0n1.out‚407.5634765625Sc#ªžÖ‚(:“ §·ôÒ²¦ÒË gpu.0.gpu‚91.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚54.0 gpu.0.powerWatts‚259.455' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.48499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛŽ¯›'ׂ(:œ ¶·ôÒ ÿËË network.sent‚670890 network.recv‚126748 disk.nvme0n1.in‚0 disk.md0.out‚ 10.10546875 disk.rbd0.out‚0) proc.memory.percent‚16.95871353149414& disk.nvme1c1n1.out‚744.59228515625! disk.dm-0.out‚622.04736328125 disk.dm-1.in‚0 disk.dm-1.out‚538.2958984375! disk.nvme0c0n1.out‚ 437.171875 proc.cpu.threads‚17( disk./.usagePercent‚76.0039858312565% disk./.usageGB‚1166.8511924743652 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚744.59228515625 disk.md0.in‚0 disk.dm-0.in‚0 disk.rbd0.in‚0$ memory_percent‚99.31647777557373 disk.nvme0n1.out‚ 437.171875 disk.nvme1c1n1.in‚0 disk.nvme0c0n1.in‚0( proc.memory.availableMB‚ 111.98828125! proc.memory.rssMB‚ 2778.515625 cpu‚26.06132368025782p1yžØ‚(:“ ¶·ôÒãÄüË gpu.0.gpu‚54.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚54.0 gpu.0.powerWatts‚265.984' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.66133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”ýM0Ù+ _wandbruntime‚321  _runtime‚321 œ‡ÆÚÀ ' lr-AdamW/pg2‚0.00017245838826764344 trainer/global_step‚649 % _timestamp‚1.7844868444570396e+09 _runtime‚ 321.032071631 _step‚48 ' lr-AdamW/pg1‚0.00017245838826764344´HRþÊÛÄ ' lr-AdamW/pg1‚0.00017245838826764344 ' lr-AdamW/pg2‚0.00017245838826764344 trainer/global_step‚649 % _timestamp‚1.7844868444570396e+09 _runtime‚ 321.032071631 _step‚480-´Z0Ü+ _wandbruntime‚321  _runtime‚321ÜP‹¸'ݲO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 605014586880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚484 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚48244655412736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340577792 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339921458 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6449787 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48849652481024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538662400 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129184256 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239511 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 144502272 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚47744816632560 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358096384 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3303720 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153608704 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 624755580928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 _runtime‚ 321.034404264 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚48849652481024 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚48244655412736 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚48349086073656 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4880282 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚38303520175616 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11330071 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11320333 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚48850010577408 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214286336 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6440051 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4954228 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚828 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚49 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214286336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6440051 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚48244637894144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4880284 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 605355164672 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 624884765184 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚48348729112838 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6449787 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 603912480278 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚48850010577408 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4880284 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1650508 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚38928251332096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚47744833671920 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 % _timestamp‚1.7844868444578612e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351•ø9!Þ¶O \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚48244655412736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340577792 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339921458 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6449787 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48849652481024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538662400 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 129184256 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239511 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 144502272 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚47744816632560 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358096384 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3303720 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153608704 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 624755580928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 _runtime‚ 321.034404264 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚48849652481024 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚48244655412736 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚48349086073656 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4880282 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚38303520175616 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11330071 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11320333 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚48850010577408 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214286336 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6440051 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4954228 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚828 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚49 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214286336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6440051 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚48244637894144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4880284 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 605355164672 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 624884765184 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚48348729112838 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6449787 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 603912480278 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚48850010577408 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4880284 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1650508 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚38928251332096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚47744833671920 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 % _timestamp‚1.7844868444578612e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 605014586880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚484 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2591‚©šá0ß+ _wandbruntime‚321  _runtime‚321÷ÑO Â&à¼M UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538662400 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4887786 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 605944498176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6449926 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚47818242715040 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6459676 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚649 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4887786 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6459676 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6449926 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚38362372083712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340584960 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11337710 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4961831 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚48318829663232 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4887784 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3308803 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚48423440379484 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 604840703514 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚50 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3308801 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1653030 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚48924774161408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚484 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339921570 _runtime‚ 321.407082885 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340584960 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 605944498176 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚828 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11347462 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356960930 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4887784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚48318847181824 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚48924774161408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚484 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339921570 _runtime‚ 321.407082885 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340584960 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 605944498176 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚828 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11347462 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356960930 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4887784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚48318847181824 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538662400 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4887786 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 605944498176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6449926 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚47818242715040 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6459676 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚649 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4887786 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6459676 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6449926 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚38362372083712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340584960 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11337710 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4961831 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚48318829663232 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4887784 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3308803 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚48423440379484 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 604840703514 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚502§%2h0â+ _wandbruntime‚321  _runtime‚321\½ˆiþãø .loss/FBt1/detection_loss‚5.438677593494097 /loss/FBt2/detection_loss‚3.1179069489240727 0loss/FRt1/assignment_loss‚3.2281245559618617 1loss/SRqqt2/assignment_loss‚1.472788767672024 1loss/SRqqt1/detection_loss‚0.9622597926524264 trainer/global_step‚649 _step‚51 /loss/FRt2/assignment_loss‚4.018815518021584 2loss/SRqqt1/assignment_loss‚1.3552256816127353 0loss/FBt2/assignment_loss‚0.5390639779807922 /loss/FRt2/detection_loss‚0.4484583509773074 %loss/total_loss‚24.94873529835283 epoch‚0 _runtime‚ 321.408640178 0loss/FBt1/assignment_loss‚0.2375890916459354 /loss/FRt1/detection_loss‚1.3974841277417984 0loss/SRqqt2/detection_loss‚2.732340891668195 % _timestamp‚1.7844868448337479e+09¢¶ÎÔäü .loss/FBt1/detection_loss‚5.438677593494097 /loss/FBt2/detection_loss‚3.1179069489240727 0loss/FRt1/assignment_loss‚3.2281245559618617 1loss/SRqqt2/assignment_loss‚1.472788767672024 1loss/SRqqt1/detection_loss‚0.9622597926524264 trainer/global_step‚649 _step‚51 /loss/FRt2/assignment_loss‚4.018815518021584 2loss/SRqqt1/assignment_loss‚1.3552256816127353 0loss/FBt2/assignment_loss‚0.5390639779807922 /loss/FRt2/detection_loss‚0.4484583509773074 %loss/total_loss‚24.94873529835283 epoch‚0 _runtime‚ 321.408640178 0loss/FBt1/assignment_loss‚0.2375890916459354 /loss/FRt1/detection_loss‚1.3974841277417984 0loss/SRqqt2/detection_loss‚2.732340891668195 % _timestamp‚1.7844868448337479e+093Ì•ÈE1å‚(:¦ Å·ôÒãŠòË disk.nvme1n1.in‚0 disk.md0.in‚0$ memory_percent‚99.31905269622803) proc.memory.percent‚16.95871353149414 network.recv‚130264) disk./.usagePercent‚76.00399477632092$ disk.nvme1n1.out‚745.23388671875 disk.rbd0.in‚0 proc.cpu.threads‚17 network.sent‚687051 disk.nvme0c0n1.in‚0 disk.nvme1c1n1.in‚0 disk.md0.out‚ 10.40234375! disk.dm-0.out‚630.23876953125( proc.memory.availableMB‚ 111.56640625! proc.memory.rssMB‚ 2778.515625 cpu‚26.077502121229056% disk./.usageGB‚1166.8513298034668% disk.nvme0c0n1.out‚445.4462890625& disk.nvme1c1n1.out‚745.23388671875 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚538.3701171875 disk.rbd0.out‚0 disk.nvme0n1.in‚0# disk.nvme0n1.out‚445.4462890625aºûgžæ‚(:“ Å·ôÒ³±«Ì gpu.0.gpu‚58.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚54.0 gpu.0.powerWatts‚263.559' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.85300000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x²g!Cç‚(:¸ Ô·ôÒ¢æôË disk.dm-0.in‚0.328125! disk.dm-1.out‚539.00341796875! proc.memory.rssMB‚ 2778.515625) disk./.usagePercent‚76.05688520293926 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.328125& disk.nvme0c0n1.out‚1277.9130859375( proc.memory.availableMB‚ 111.26171875) proc.memory.percent‚16.95871353149414 cpu‚26.1035151364804 network.recv‚134526$ disk.nvme0n1.out‚1277.9130859375 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚750.6064453125" disk.dm-0.out‚1465.16162109375$ memory_percent‚99.32091236114502 proc.cpu.threads‚17 disk.nvme1n1.in‚0 network.sent‚721161$ disk./.usageGB‚1167.663330078125 disk.rbd0.out‚0 disk.nvme0n1.in‚0.328125 disk.md0.out‚ 11.515625 disk.dm-1.in‚0# disk.nvme1n1.out‚750.6064453125 disk.md0.in‚0M×õOžè‚(:“ Ô·ôÒÓœ§Ì gpu.0.gpu‚94.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚54.0 gpu.0.powerWatts‚261.131' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.04366666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K°(0é+ _wandbruntime‚346  _runtime‚3462vƒÅê¿ ' lr-AdamW/pg1‚0.00018574485885837098 ' lr-AdamW/pg2‚0.00018574485885837098 trainer/global_step‚699 $ _timestamp‚1.784486869877955e+09 _runtime‚ 346.452917554 _step‚529Z[íÉëà ' lr-AdamW/pg1‚0.00018574485885837098 ' lr-AdamW/pg2‚0.00018574485885837098 trainer/global_step‚699 $ _timestamp‚1.784486869877955e+09 _runtime‚ 346.452917554 _step‚524©k\?0ì+ _wandbruntime‚346  _runtime‚346LÒ@Ûí±O IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚41920227979776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5334934 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 650355545286 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚52606126853632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1777448 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340577792 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51954243867136 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 672774575616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51954226348544 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5255382 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚41920086953984 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538662400 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5335220 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚52605768757248 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 651882986496 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12191035 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5255382 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚484 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5255384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 651542408704 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3557770 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 651882986496 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚51416120756560 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 _step‚53 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚828 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 $ _timestamp‚1.784486869878898e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 651542408704 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6945389 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚51416137795920 trainer/global_step‚699 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3557772 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339921458 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358096384 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚51954243867136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41247312378368 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5255384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41247336802816 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214286336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51954226348544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 116601344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52066476301846 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0¼?ý»'îµO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚828 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 $ _timestamp‚1.784486869878898e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 651542408704 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6945389 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚51416137795920 trainer/global_step‚699 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3557772 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339921458 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358096384 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚51954243867136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41247312378368 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5255384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41247336802816 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214286336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51954226348544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 116601344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52066476301846 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚41920227979776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5334934 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 650355545286 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚52606126853632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1777448 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340577792 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51954243867136 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 672774575616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51954226348544 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5255382 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚41920086953984 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538662400 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5335220 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚52605768757248 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 651882986496 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12191035 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5255382 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚484 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5255384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 651542408704 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3557770 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 651882986496 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚51416120756560 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 _step‚535vׄ0ï+ _wandbruntime‚346  _runtime‚346S9ÎÌÁ&ð»M MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6945528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚52681248541184 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3562853 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356960930 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚52140830607562 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 651623690092 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6945528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340584960 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41980069004800 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5262884 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5262886 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚52141187568492 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537999223 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚52028418117632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚52028435636224 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 673855869440 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41306188710912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 652472320000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538662400 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1779725 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚41979927986176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 _step‚54 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5262886 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12218164 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3562851 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 116594176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214286336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 652812904960 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538662400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚484 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚51489563878400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 trainer/global_step‚699 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339921570 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1779988 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 _runtime‚ 346.823669473 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6955278 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 141018624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41306213135360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51489546839040 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚52028418117632 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 652812904960 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 673739275264 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358103552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12208412 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5342576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 144502272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚52681248541184;5•þU#ñ¿M MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6955278 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚828 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 _runtime‚ 346.823669473 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6955278 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 141018624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41306213135360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51489546839040 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚52028418117632 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 652812904960 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 673739275264 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358103552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12208412 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5342576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 144502272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚52681248541184 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6945528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚52681248541184 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3562853 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356960930 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚52140830607562 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 651623690092 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6945528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340584960 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41980069004800 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5262884 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5262886 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚52141187568492 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537999223 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚52028418117632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚52028435636224 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 673855869440 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41306188710912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 652472320000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538662400 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1779725 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚41979927986176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 _step‚54 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5262886 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12218164 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3562851 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 116594176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214286336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 652812904960 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538662400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚484 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚51489563878400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 trainer/global_step‚699 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.larg“åFpe_pool.current‚24424448 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339921570 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1779988 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 ODeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 371.701945094 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7440991 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 698410808320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 % _timestamp‚1.7844868951270115e+09 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚56362243129856 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚56361885033472 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3811822 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538662400 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358096384 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚56362243129856 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚56361885033472 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚55663832321536 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356960818 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚44911881537024 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358096384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 720888007168 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚55784580451672 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7431255 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 720752531456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚55087424880560 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339921458 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239511 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 697138531752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚55663814802944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537999223 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214286336 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚55784223490854 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚55663832321536 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚828 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7440991 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3811820 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚484 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 135475712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214286336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340577792 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538662400 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚55087441919920 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5630484 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1904066þ|¼'ü¶O [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚55087424880560 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339921458 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239511 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 697138531752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚55663814802944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 537999223 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214286336 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚55784223490854 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚55663832321536 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚828 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7440991 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3811820 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚484 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 135475712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214286336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340577792 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538662400 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚55087441919920 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5630484 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1904066 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 698070230528 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13071475 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340577792 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13061737 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5715886 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1904357 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 371.701945094 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7440991 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 698410808320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 % _timestamp‚1.7844868951270115e+09 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚56362243129856 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚56361885033472 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3811822 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538662400 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358096384 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚56362243129856 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚56361885033472 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚55663832321536 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356960818 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚44911881537024 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358096384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 720888007168 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚55784580451672 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7431255 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 720752531456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚09+0ý+ _wandbruntime‚372  _runtime‚372¸l©þ¼M ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3816903 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚44971873346560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7441130 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚55738006572032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 699340726784 _step‚58 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5637984 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚56437364817408 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 699000141824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214286336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13088866 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5637986 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 699340726784 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538662400 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 trainer/global_step‚749 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13079114 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157795840 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5723498 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚55738024090624 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚56437364817408 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚56437006713856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339921570 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 697726833530 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537999223 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚55858577796570 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358103552 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340584960 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5637986 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚55858934757500 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1906885 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 698066755100 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5723788 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚484 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356960930 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13088866 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340584960 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚55160850963040 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚44971715550720 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚44250005338112 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133371392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚ 372.071698112 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7441130 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13079114 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259ßͽÆ&ÿÀM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356960930 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13088866 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340584960 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚55160850963040 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚44971715550720 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚44250005338112 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133371392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚ 372.071698112 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7441130 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13079114 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3816903 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚44971873346560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7441130 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚55738006572032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 699340726784 _step‚58 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5637984 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚56437364817408 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 699000141824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214286336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13088866 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5637986 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 699340726784 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538662400 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 trainer/global_step‚749 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13079114 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157795840 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5723498 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚55738024090624 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚56437364817408 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚56437006713856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339921570 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 697726833530 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537999223 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚55858577796570 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358103552 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340584960 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5637986 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚55858934757500 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1906885 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 698066755100 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5723788 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚484:âò²´0€+ _wandbruntime‚372  _runtime‚372gèÀÿþø 2loss/SRqqt2/assignment_loss‚1.5957909589575499 1loss/FBt1/assignment_loss‚0.17268177544207383 0loss/FBt2/assignment_loss‚0.8043602504722003 .loss/FBt1/detection_loss‚2.878244099523671 trainer/global_step‚749 _runtime‚ 372.072279024 1loss/SRqqt2/detection_loss‚2.5511835104676743 _step‚59 /loss/FRt2/assignment_loss‚4.573881824927149 /loss/FRt1/detection_loss‚1.2801554535492754 .loss/FBt2/detection_loss‚2.057159165541331 /loss/FRt2/detection_loss‚0.4754200472056753 1loss/SRqqt1/detection_loss‚0.8631467414946974 %loss/total_loss‚21.32922974855815 epoch‚0 $ _timestamp‚1.784486895497567e+09 0loss/FRt1/assignment_loss‚2.8488566146041805 1loss/SRqqt1/assignment_loss‚1.228349306372672 ¥I‚ü 1loss/SRqqt1/assignment_loss‚1.228349306372672 2loss/SRqqt2/assignment_loss‚1.5957909589575499 1loss/FBt1/assignment_loss‚0.17268177544207383 0loss/FBt2/assignment_loss‚0.8043602504722003 .loss/FBt1/detection_loss‚2.878244099523671 trainer/global_step‚749 _runtime‚ 372.072279024 1loss/SRqqt2/detection_loss‚2.5511835104676743 _step‚59 /loss/FRt2/assignment_loss‚4.573881824927149 /loss/FRt1/detection_loss‚1.2801554535492754 .loss/FBt2/detection_loss‚2.057159165541331 /loss/FRt2/detection_loss‚0.4754200472056753 1loss/SRqqt1/detection_loss‚0.8631467414946974 %loss/total_loss‚21.32922974855815 epoch‚0 $ _timestamp‚1.784486895497567e+09 0loss/FRt1/assignment_loss‚2.8488566146041805;ÜYïÛ<ƒ‚(:± ò·ôÒã•Ì cpu‚26.153370973422255) disk./.usagePercent‚76.30361020613563& disk.nvme1c1n1.out‚817.18017578125! disk.dm-0.out‚1912.2705078125 disk.dm-1.in‚0 disk.rbd0.out‚0 network.sent‚785319% disk./.usageGB‚1171.4511756896973' disk.nvme0c0n1.out‚1663.80224609375$ disk.nvme1n1.out‚817.18017578125$ memory_percent‚99.32546615600586& proc.memory.availableMB‚ 110.515625 proc.cpu.threads‚17 disk.nvme0n1.in‚4.3125% disk.nvme0n1.out‚1663.80224609375 disk.nvme1n1.in‚0 disk.md0.out‚ 13.84765625 disk.dm-0.in‚4.3125 disk.dm-1.out‚539.5625) proc.memory.percent‚16.95871353149414 network.recv‚142238 disk.nvme0c0n1.in‚4.3125 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.rbd0.in‚0! proc.memory.rssMB‚ 2778.515625˜#ª‡›„‚(: ò·ôÒ›²±Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚52.0 gpu.0.powerWatts‚117.98' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.32666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ×@…‚(:µ ¸ôÒÔ®ñË) disk./.usagePercent‚76.55676397743841 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚1096.90771484375 disk.md0.out‚ 14.2890625 disk.dm-0.out‚8069.701171875 disk.dm-1.in‚0) proc.memory.percent‚16.95871353149414 network.sent‚821398 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4.3125 disk.md0.in‚0! proc.memory.rssMB‚ 2778.515625 network.recv‚146556$ disk./.usageGB‚1175.337718963623 disk.rbd0.out‚0' disk.nvme0c0n1.out‚7543.06982421875 disk.nvme0n1.in‚4.3125 disk.dm-0.in‚4.3125 disk.dm-1.out‚ 540.1640625( proc.memory.availableMB‚ 110.70703125% disk.nvme0n1.out‚7543.06982421875' disk.nvme1c1n1.out‚1096.90771484375 disk.nvme1n1.in‚0$ memory_percent‚99.32429790496826 cpu‚26.168736178491528 proc.cpu.threads‚17âš'“†‚(:ˆ ¸ôÒ†”¯Ì gpu.0.gpu‚81.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚54.0 gpu.0.powerWatts‚251.181' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚83.727 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŠkÒ0‡+ _wandbruntime‚396  _runtime‚3963O^ƈÀ ' lr-AdamW/pg1‚0.00021231780003982603 ' lr-AdamW/pg2‚0.00021231780003982603 trainer/global_step‚799 % _timestamp‚1.7844869201595817e+09 _runtime‚ 396.735679813 _step‚60ÓRU{Ê‰Ä ' lr-AdamW/pg2‚0.00021231780003982603 trainer/global_step‚799 % _timestamp‚1.7844869201595817e+09 _runtime‚ 396.735679813 _step‚60 ' lr-AdamW/pg1‚0.00021231780003982603<ò0Š+ _wandbruntime‚396  _runtime‚396•1ua·'‹±O F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚60118359406080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 trainer/global_step‚799 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 743581596760 $ _timestamp‚1.784486920161416e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 744938630144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚828 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4065872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 768891301888 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7926857 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚60118359406080 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 768768409088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚60118001309696 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538662400 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6005584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4065870 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147317248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358096384 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13942177 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚484 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2031999 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7926857 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239511 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340577792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356960818 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340577792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358096384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚61 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚47134945632768 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚59373420775936 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 144502272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339921458 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 743241675302 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚59373403257344 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚58758746043920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538662400 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 _runtime‚ 396.736299454 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚60118001309696 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 744938630144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6097597 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122892800 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214286336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6097871 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2031727 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚58758729004560 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934`N¬b7ŒµO VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚59373403257344 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214286336 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6005582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7926857 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030239511 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340577792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356960818 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340577792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358096384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚61 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚47134945632768 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚59373420775936 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 144502272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339921458 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 743241675302 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚59373403257344 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚58758746043920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538662400 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 _runtime‚ 396.736299454 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚60118001309696 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 744938630144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6097597 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122892800 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214286336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6097871 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceEmÔ}„StatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2031727 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚58758729004560 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚60118359406080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 trainer/global_step‚799 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 743581596760 $ _timestamp‚1.784486920161416e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 744938630144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚828 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4065872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 768891301888 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7926857 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚60118359406080 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 768768409088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚60118001309696 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538662400 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6005584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4065870 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147317248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358096384 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13942177 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚484 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2031999 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 397.09585923 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 744169898538 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6013086 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2034542 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 144502272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚47963552294912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13949816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚58832155087040 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340584960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚484 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚828 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13959568 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 744509820108 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 745527963648 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538662400 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358103552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚59447612545024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 745527963648 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4070953 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚59576681946508 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚59447595026432 % _timestamp‚1.7844869205207543e+09 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚59447595026432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149407232 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚47193846389760 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124982784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214286336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340584960 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6105495 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 769730329600 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537999223 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7946482 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13959568 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6013086 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚60193481093632 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚47963701702144 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6013084 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6105207 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4070951 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7936732 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚59447612545024 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 745868548608 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7946482 _step‚62 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚59576324985578 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚58832172126400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 769855312384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356960930¦å*Å&¿M I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356960930 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6013084 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7936732 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2034256 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13949816 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚47193821965312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 397.09585923 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030239511 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 744169898538 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6013086 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2034542 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 144502272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚47963552294912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13949816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚58832155087040 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340584960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚484 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚828 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13959568 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 744509820108 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 745527963648 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538662400 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358103552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚59447612545024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 745527963648 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4070953 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚59576681946508 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚59447595026432 % _timestamp‚1.7844869205207543e+09 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358103552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚59447595026432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149407232 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚47193846389760 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214286336 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124982784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214286336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340584960 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6105495 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 769730329600 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 537999223 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7946482 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13959568 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6013086 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚60193481093632 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚47963701702144 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6013084 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6105207 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4070951 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7936732 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚59447612545024 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 745868548608 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7946482 _step‚62 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚59576324985578 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚58832172126400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 769855312384>+D \0+ _wandbruntime‚397  _runtime‚3972²ë­ü‘ö 1loss/FBt1/assignment_loss‚0.13854472907549584 -loss/FRt1/detection_loss‚1.33804298638234 1loss/SRqqt1/detection_loss‚0.8623333431221637 0loss/SRqqt2/detection_loss‚2.797100708762554 /loss/FBt2/detection_loss‚14.342672342061997 .loss/FRt2/detection_loss‚0.443288141646128 % _timestamp‚1.7844869205217142e+09 %loss/total_loss‚34.56266983417256 /loss/FRt1/assignment_loss‚2.754558079625761 /loss/FRt2/assignment_loss‚4.592521041789383 .loss/FBt1/detection_loss‚4.509193544784863 trainer/global_step‚799 2loss/SRqqt1/assignment_loss‚1.0899185168385592 1loss/SRqqt2/assignment_loss‚1.500709507796731 1loss/FBt2/assignment_loss‚0.19378689228658458 epoch‚0 _runtime‚ 397.096364441 _step‚63ö(ø’ú epoch‚0 _runtime‚ 397.096364441 _step‚63 1loss/FBt1/assignment_loss‚0.13854472907549584 -loss/FRt1/detection_loss‚1.33804298638234 1loss/SRqqt1/detection_loss‚0.8623333431221637 0loss/SRqqt2/detection_loss‚2.797100708762554 /loss/FBt2/detection_loss‚14.342672342061997 .loss/FRt2/detection_loss‚0.443288141646128 % _timestamp‚1.7844869205217142e+09 %loss/total_loss‚34.56266983417256 /loss/FRt1/assignment_loss‚2.754558079625761 /loss/FRt2/assignment_loss‚4.592521041789383 .loss/FBt1/detection_loss‚4.509193544784863 trainer/global_step‚799 2loss/SRqqt1/assignment_loss‚1.0899185168385592 1loss/SRqqt2/assignment_loss‚1.500709507796731 1loss/FBt2/assignment_loss‚0.19378689228658458?Z½úC“‚(:¸ ¸ôÒáçË network.sent‚837601% disk./.usageGB‚1175.2733764648438% disk.nvme1n1.out‚1392.04736328125 disk.md0.in‚0 disk.dm-0.in‚4.3125 disk.nvme0c0n1.in‚4.3125' disk.nvme0c0n1.out‚8495.84326171875 disk.nvme0n1.in‚4.3125 network.recv‚150153 disk.nvme1n1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚540.5302734375 disk.rbd0.in‚0% disk.nvme0n1.out‚8495.84326171875 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚1392.04736328125) disk./.usagePercent‚76.55257296628196 disk.dm-0.out‚9316.193359375' proc.memory.availableMB‚ 67.42578125 proc.cpu.threads‚17 disk.md0.out‚ 14.79296875 disk.rbd0.out‚0# memory_percent‚99.5884656906128 proc.memory.rssMB‚ 2821.65625* proc.memory.percent‚17.222023010253906 cpu‚26.239689826960745´Ôy–Ÿ”‚(:” ¸ôÒ—õšÌ gpu.0.gpu‚97.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚121.111' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.370333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0" ‹J•bE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP90òH–bC Purity/1FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPÌ ÿ‚N—bI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPkP??H˜bC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP_)ŽµH™bC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPÔ[t,JšbE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPÈÏêˆH›bC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPJaW˜JœbE "Purity/2FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPøð&fHbC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPd´WJžbE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPݺRJNŸbI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPÑû€lJ bE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPEŽÔÁJ¡bE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP­ì©J¢bE "Purity/1FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPq ñ“N£bI &Purity/1FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP•±H¤bC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP÷ĶN¥bI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPû·Äœ;¦b6 jet/accuracy_2_of_3"trainer/global_step2:JPRn,H§bC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP?;¨b6 jet/accuracy_2_of_5"trainer/global_step2:JP7ÌxéJ©bE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP=ŽŸJªbE "Purity/0FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPþdØoN«bI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPÍ(,¥H¬bC Purity/1FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP7 PIJ­bE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPê•@®b; particle/accuracy_1_of_2"trainer/global_step2:JPý¸>XH¯bC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP yJ°bE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¢¬~N±bI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP#zŠ;H²bC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPXš‡N³bI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPˆy gN´bI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP›¹ËKHµbC Purity/*FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP1ê¬vJ¶bE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP}rû©H·bC Purity/0FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPt†fJ¸bE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP7 ¯èJ¹bE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP€ /JºbE "Purity/*FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP©®SìJ»bE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPË.,–;¼b6 jet/accuracy_1_of_3"trainer/global_step2:JPÁN½bI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPâÐH¾bC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP»7š¤J¿bE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPM@Àb; particle/accuracy_1_of_1"trainer/global_step2:JPzgePHÁbC Purity/1FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPn¢ùhHÂbC Purity/2FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPÝ?å¿HÃbC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP)zKHÄbC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPúR¯˜HÅbC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP‡/ÆJÆbE "Purity/1FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPÃ÷†´HÇbC Purity/1FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPOáÉ¿HÈbC Purity/1FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP††/JÉbE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPv¿ÑJÊbE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÖ[;Ëb6 jet/accuracy_1_of_2"trainer/global_step2:JP,ÑròJÌbE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPÐJÍbE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPÓÌúJÎbE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPË ŽJÏbE "Purity/*FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP…B|ÒJÐbE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÓç^NÑbI &Purity/2FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPE;Þ9Òb4 particle/accuracy"trainer/global_step2:JPrs*ûHÓbC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP„žNÔbI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPÝ[ûJÕbE "Purity/*FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÅöÀ'HÖbC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPr7ƒŸH×bC Purity/2FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP¢ÓXÕJØbE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÃ$Ô•NÙbI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP¹K›JÚbE "Purity/0FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP-×,JÛbE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPgL òNÜbI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPSt³HHÝbC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPÊB_HÞbC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP,ì µHßbC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPN_QÙàbE "Purity/2FRt1SRqqtv Å2*FBt/event_purity"trainer/global_step2:JPï£6;áb6 jet/accuracy_4_of_4"trainer/global_step2:JPŠ@jÀJâbE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP0êò“NãbI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPi< BNäbI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPo¿ý<åb7 particle/sensitivity"trainer/global_step2:JP¾f›?JæbE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP?z0òNçbI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPyÜcHèbC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPÓüŽJébE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPd\ò†JêbE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP3ÙØ%JëbE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP –‹ˆJìbE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP5íJíbE "Purity/*FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP“?áYJîbE "Purity/0FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPå€ã;ïb6 jet/accuracy_1_of_5"trainer/global_step2:JPÅ?B JðbE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPÊ ÿNñbI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP¹·×KNòbI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP9Ç×—JóbE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP²¦sƒHôbC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPêâ4•JõbE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¶Bmâ@öb; particle/accuracy_3_of_3"trainer/global_step2:JPÐÓ†*J÷bE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP—µz¦JøbE "Purity/1FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPÀª WHùbC Purity/*FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPe<É„NúbI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP“6“;ûb6 jet/accuracy_2_of_4"trainer/global_step2:JP ×oNübI &Purity/0FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP²OJýbE "Purity/1FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP=¤&‚JþbE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP5WßHÿbC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPžäzÔJ€bE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPð›ÕHbC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP(Br¡H‚bC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP¼èÒR@ƒb; particle/accuracy_3_of_4"trainer/global_step2:JPs²VN„bI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPÆ@H…bC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPyJ‘@†b; particle/accuracy_4_of_5"trainer/global_step2:JPßÇ€J‡bE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPç"SäHˆbC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPX‹G J‰bE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP‡kHøNŠbI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP/0:ÌH‹bC Purity/1FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPZVªONŒbI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP ‚¦@b; particle/accuracy_3_of_5"trainer/global_step2:JPT:ïJŽbE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPª²ÁÐJbE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPz†JbE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPøFBH‘bC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPë%~3N’bI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP*TR0H“bC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPWÚ%J”bE "Purity/*FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPÕzZN•bI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPÛœÿ}H–bC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP½CcqH—bC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPá0Ìß8˜b3 particle/f_score"trainer/global_step2:JPˆä©¸H™bC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP®Ñ»ìNšbI &Purity/*FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP‡…»óN›bI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPìkú#NœbI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPÅÉÛQJbE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPw,aNžbI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP—fö;Ÿb6 jet/accuracy_1_of_1"trainer/global_step2:JP³{ÐÇH bC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPjèáAH¡bC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP–Ü¿J¢bE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPòáuH£bC Purity/2FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP£úH¤bC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP¬¼ÒH¥bC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPé^* J¦bE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP¥¢ž?N§bI &Purity/1FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPĈ k;¨b6 jet/accuracy_3_of_4"trainer/global_step2:JP9±Ñ¢H©bC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP°€¼c;ªb6 jet/accuracy_1_of_4"trainer/global_step2:JP¬†ÑâN«bI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP.ž N¬bI &Purity/2FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP»Ð”N­bI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPaâ’ð@®b; particle/accuracy_2_of_2"trainer/global_step2:JP n,{@¯b; particle/accuracy_2_of_4"trainer/global_step2:JPC_;°b6 jet/accuracy_3_of_5"trainer/global_step2:JP붑N±bI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPõxÕH²bC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPÒýN³bI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP‘ð»ÂH´bC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPÐQ&‚JµbE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP—À¹J¶bE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP\b¡J·bE "Purity/*FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPD9J¸bE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPTíôH¹bC Purity/2FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPñ/«HºbC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPÊf¹J»bE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP¨¯ÆG¼bB validation_average_jet_accuracy"trainer/global_step2:JPÅ €þJ½bE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP^{NïJ¾bE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP ‡^­J¿bE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP´ž<ÃNÀbI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP–)­HÁbC Purity/2FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPSöÒNÂbI &Purity/2FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÒòŽtJÃbE "Purity/1FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP§þNÄbI &Purity/0FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPüTcVJÅbE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP\YkH;Æb6 jet/accuracy_2_of_2"trainer/global_step2:JPõûx_@Çb; particle/accuracy_1_of_5"trainer/global_step2:JPdÁÃqNÈbI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPžÑÆúHÉbC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPÎå%¹JÊbE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP(¢fNËbI &Purity/1FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP·jJÌbE "Purity/2FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP ¨¾ŠNÍbI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP<œAÃ@Îb; particle/accuracy_1_of_4"trainer/global_step2:JPJa¬JÏbE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPÄ„bNÐbI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP:Ž;NNÑbI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPå>wNÒbI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP“"÷¸JÓbE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPò‘°JÔbE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP wåIJÕbE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPËÿå@Öb; particle/accuracy_5_of_5"trainer/global_step2:JP®uþN×bI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP¿‚#NØbI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPp~ÄóHÙbC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPϨôôJÚbE "Purity/0FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPl}”HÛbC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPÓ+HÜbC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPÐy›¼HÝbC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP¤CmÊNÞbI &Purity/*FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP,úš3NßbI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPv­7×HàbC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP™æÀüHábC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPYº"ˆ@âb; particle/accuracy_2_of_3"trainer/global_step2:JP§™UYJãbE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP܆ñHäbC Purity/*FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP›èiÈHåbC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPÉ~NæbI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP¹š£µJçbE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPǦéáHèbC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPª¿òJ@éb; particle/accuracy_2_of_5"trainer/global_step2:JPÀõ,NêbI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPˆ PÔHëbC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP\ùXçJìbE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPBee;íb6 jet/accuracy_5_of_5"trainer/global_step2:JPÄ+AHîbC Purity/0FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP× þ’HïbC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPçÎ2JðbE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPÃá^ÉHñbC Purity/1FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP£¶JòbE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP·ówHóbC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP%bŒóJôbE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP·WmÓHõbC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPÅR’¹JöbE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP ¯|H÷bC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP¤±¢àHøbC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP{D±HùbC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPÂ7-kNúbI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPø-è%JûbE "Purity/2FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP6q«JübE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP}]~/HýbC Purity/0FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP¢&ÕãHþbC Purity/1FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP.ÂÏúJÿbE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPnÍEN€bI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP3Ù JbE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPûùØý;‚b6 jet/accuracy_3_of_3"trainer/global_step2:JPúo JƒbE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPX‹€J„bE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP@C÷H…bC Purity/1FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPërEH†bC Purity/1FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPŽ5ÑJ‡bE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPÄë JˆbE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP;k’ŸH‰bC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP©L~fJŠbE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÎÞ›H‹bC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP¢çÓHŒbC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPÖa/2JbE "Purity/0FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPF/ƒïJŽbE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP úÔ­HbC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP%@ÿ¨JbE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPBò?H‘bC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP|{-ßH’bC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP¿Æ J“bE "Purity/0FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP“Ox[J”bE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP–ªN•bI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP¤ UÃJ–bE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPíáI–H—bC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP¹GèÏH˜bC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP8¦hÈJ™bE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP™‹¸=HšbC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPC¡eeH›bC Purity/2FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPùÁœêJœbE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP¡å¹/NbI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPCONžbI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPŒíŽNŸbI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPòæz5J bE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÝ6‚ÂJ¡bE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPoxbw@¢b; particle/accuracy_1_of_3"trainer/global_step2:JP"5~iJ£bE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPHÐôH¤bC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPΧLÜN¥bI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPDããN¦bI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPˆ’¸N§bI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP:e_n@¨b; particle/accuracy_4_of_4"trainer/global_step2:JP“»;©b6 loss/val_total_loss"trainer/global_step2:JP4ÞJªbE "Purity/*FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPDÈÎBJ«bE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP-U4N¬bI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPÌN­bI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPùN®bI &Purity/1FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP×-’}H¯bC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPæ)-J°bE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP(ƒ"Þ<±b7 particle/specificity"trainer/global_step2:JP‹à;²b6 jet/accuracy_4_of_5"trainer/global_step2:JPê3WN³bI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPçá¶LN´bI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPðÍ’Ï0µ+ _wandbruntime‚417  _runtime‚417¡3\ˆ=¶½{ ("Purity/2FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3086419701576233 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.22346827387809753 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.14006802439689636 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 (particle/accuracy‚0.8010603785514832 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961348176002502 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.1644854098558426 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9954367876052856 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.04388615861535072 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.1532752811908722 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.06643540412187576 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.2748746871948242 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.40253785252571106 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_4‚0 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.16551442444324493 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 _runtime‚ 417.03995747 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *particle/sensitivity‚0.512041449546814 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.455205500125885 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15875330567359924 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.469058096408844 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.24304836988449097 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.07039839774370193 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0416666679084301 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.13617663085460663 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.41454315185546875 jet/accuracy_1_of_5‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.006722699850797653 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.008043721318244934 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.23698082566261292 /particle/accuracy_3_of_3‚0.9989849328994751 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15550558269023895 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 +jet/accuracy_2_of_4‚0.13328345119953156 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.06643540412187576 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.23738649487495422 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7756974101066589 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.04770047962665558 /particle/accuracy_3_of_4‚0.9943783283233643 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.07258599996566772 particle/accuracy_4_of_5‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.0849381536245346 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.3943154513835907 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.16628259420394897 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5914198756217957 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.172155499458313 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.42130228877067566 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ( Purity/1FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.06371450424194336 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9865072965621948 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.5250118970870972 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8166666030883789 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5321213603019714 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +jet/accuracy_1_of_1‚0.11349426209926605 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.25 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2533024549484253 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9877657890319824 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5455120801925659 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ,jet/accuracy_3_of_4‚0.007407407741993666 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9616227149963379 *jet/accuracy_1_of_4‚0.5595901608467102 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_4‚1 jet/accuracy_3_of_5‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.2521076202392578 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.13808506727218628 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.1478983759880066 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.24573750793933868 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9787704944610596 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.08042050153017044 6validation_average_jet_accuracy‚0.1596258282661438 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.06712812185287476 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.12538617849349976 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12071245908737183 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.21096490323543549 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.16223721206188202 ,jet/accuracy_2_of_2‚0.056187838315963745 particle/accuracy_1_of_5‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9430583715438843 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.22163717448711395 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.002150925574824214 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_4‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6800000071525574 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.16551442444324493 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.24166667461395264 particle/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.25273481011390686 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.35227271914482117 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1637844741344452 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2554122507572174 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1123303696513176 particle/accuracy_2_of_3‚1 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.19914528727531433 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9958924651145935 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9348069429397583 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.004857082385569811 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9408119320869446 particle/accuracy_2_of_5‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.29239922761917114 +"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.25 jet/accuracy_5_of_5‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9780001640319824 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9943616986274719 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.05259110778570175 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9860460162162781 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.04914453998208046 . Purity/2FRt*SRqqt1FBt/FRt_purity‚ 0.1015625 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3214285671710968 trainer/global_step‚800 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8035714030265808 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.07046429067850113 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.20256826281547546 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 /"Purity/2FRt*SRqqt1FBt/event_purity‚0.015625 epoch‚0 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.015032296068966389 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9860220551490784 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.42660221457481384 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.12188313156366348 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.017965510487556458 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7026143670082092 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.22323504090309143 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2764231264591217 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9981684684753418 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.004977370146661997 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.12111042439937592 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.06450141966342926 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2023347169160843 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.878746509552002 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.24166667461395264 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.2982455790042877 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3214285671710968 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.2142857164144516 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9715217351913452 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.16294541954994202 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004637820180505514 ( Purity/*FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978310465812683 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.014595873653888702 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.4193327724933624 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 % _timestamp‚1.7844869404645352e+09 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.14895083010196686 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6800000071525574 particle/accuracy_1_of_3‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 /particle/accuracy_4_of_4‚0.9304422736167908 *loss/val_total_loss‚27.897751768504392 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.6552563905715942 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8264936804771423 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7942681312561035 +particle/specificity‚0.8801054954528809 jet/accuracy_4_of_5‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.22652317583560944 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16003063321113586 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24406830966472626 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.26444071531295776 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0024049195926636457 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1123303696513176 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.006820818409323692 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.14599794149398804 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.21074095368385315 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.12031973153352737 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.18527336418628693 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 jet/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.15148130059242249 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.15361319482326508 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9905603528022766 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_2‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.1309611052274704 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9805985689163208 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.23231804370880127 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9943616986274719 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9393731951713562 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.14332592487335205 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.13808506727218628 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.10798528045415878 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.06939119100570679 *jet/accuracy_1_of_3‚0.6583575010299683 _step‚64 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.25150683522224426 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_1‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.3183760344982147 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.1458333283662796 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9368823766708374 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9944092631340027 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9764772057533264 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1561780571937561 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9987468719482422 5 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.28376504778862 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.15668076276779175 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 *jet/accuracy_1_of_2‚0B*€;.4945652484893799 ("Purity/2FRt*SRqqt2FBt/event_purity‚0ŽØdøÇ=·Á{ 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9805985689163208 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.23231804370880127 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9943616986274719 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9393731951713562 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.14332592487335205 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.13808506727218628 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.10798528045415878 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.06939119100570679 *jet/accuracy_1_of_3‚0.6583575010299683 _step‚64 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.25150683522224426 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_1‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.3183760344982147 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.1458333283662796 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9368823766708374 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9944092631340027 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9764772057533264 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1561780571937561 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9987468719482422 5 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.28376504778862 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.15668076276779175 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 *jet/accuracy_1_of_2‚0.4945652484893799 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3086419701576233 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.22346827387809753 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.14006802439689636 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 (particle/accuracy‚0.8010603785514832 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961348176002502 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.1644854098558426 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9954367876052856 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.04388615861535072 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.1532752811908722 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.06643540412187576 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.2748746871948242 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.40253785252571106 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_4‚0 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.16551442444324493 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 _runtime‚ 417.03995747 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *particle/sensitivity‚0.512041449546814 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.455205500125885 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15875330567359924 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.469058096408844 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.24304836988449097 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.07039839774370193 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0416666679084301 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.13617663085460663 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.41454315185546875 jet/accuracy_1_of_5‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.006722699850797653 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.008043721318244934 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.23698082566261292 /particle/accuracy_3_of_3‚0.9989849328994751 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15550558269023895 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 +jet/accuracy_2_of_4‚0.13328345119953156 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.06643540412187576 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.23738649487495422 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7756974101066589 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.04770047962665558 /particle/accuracy_3_of_4‚0.9943783283233643 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.07258599996566772 particle/accuracy_4_of_5‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.0849381536245346 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.3943154513835907 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.16628259420394897 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5914198756217957 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.172155499458313 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.42130228877067566 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ( Purity/1FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.06371450424194336 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9865072965621948 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.5250118970870972 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8166666030883789 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5321213603019714 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +jet/accuracy_1_of_1‚0.11349426209926605 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.25 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2533024549484253 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9877657890319824 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5455120801925659 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ,jet/accuracy_3_of_4‚0.007407407741993666 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9616227149963379 *jet/accuracy_1_of_4‚0.5595901608467102 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_4‚1 jet/accuracy_3_of_5‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.2521076202392578 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.13808506727218628 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.1478983759880066 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.24573750793933868 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9787704944610596 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.08042050153017044 6validation_average_jet_accuracy‚0.1596258282661438 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.06712812185287476 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.12538617849349976 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12071245908737183 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.21096490323543549 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.16223721206188202 ,jet/accuracy_2_of_2‚0.056187838315963745 particle/accuracy_1_of_5‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9430583715438843 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.22163717448711395 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.002150925574824214 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_4‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6800000071525574 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.16551442444324493 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.24166667461395264 particle/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.25273481011390686 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.35227271914482117 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1637844741344452 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2554122507572174 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1123303696513176 particle/accuracy_2_of_3‚1 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.19914528727531433 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9958924651145935 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9348069429397583 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.004857082385569811 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9408119320869446 particle/accuracy_2_of_5‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.29239922761917114 +"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.25 jet/accuracy_5_of_5‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9780001640319824 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9943616986274719 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.05259110778570175 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9860460162162781 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.04914453998208046 . Purity/2FRt*SRqqt1FBt/FRt_purity‚ 0.1015625 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3214285671710968 trainer/global_step‚800 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8035714030265808 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.07046429067850113 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.20256826281547546 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 /"Purity/2FRt*SRqqt1FBt/event_purity‚0.015625 epoch‚0 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.015032296068966389 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9860220551490784 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.42660221457481384 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.12188313156366348 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.017965510487556458 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7026143670082092 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.22323504090309143 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2764231264591217 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9981684684753418 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.004977370146661997 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.12111042439937592 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.06450141966342926 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2023347169160843 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.878746509552002 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.24166667461395264 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.2982455790042877 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3214285671710968 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.2142857164144516 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9715217351913452 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.16294541954994202 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004637820180505514 ( Purity/*FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978310465812683 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.014595873653888702 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.4193327724933624 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 % _timestamp‚1.7844869404645352e+09 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.14895083010196686 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6800000071525574 particle/accuracy_1_of_3‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 /particle/accuracy_4_of_4‚0.9304422736167908 *loss/val_total_loss‚27.897751768504392 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.6552563905715942 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8264936804771423 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7942681312561035 +particle/specificity‚0.8801054954528809 jet/accuracy_4_of_5‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.22652317583560944 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16003063321113586 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24406830966472626 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.26444071531295776 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0024049195926636457 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1123303696513176 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.006820818409323692 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.14599794149398804 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.21074095368385315 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.12031973153352737 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.18527336418628693 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 jet/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.15148130059242249 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.15361319482326508 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9905603528022766 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_2‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.1309611052274704 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031@h“îˆB¸‚(:· Ÿ¸ôÒâ ÆÍ# proc.memory.rssMB‚ 2837.51171875) proc.memory.percent‚17.31879711151123 cpu‚26.34699111306584$ disk.nvme0n1.out‚8505.2197265625' disk.nvme1c1n1.out‚1397.71728515625 disk.nvme1n1.in‚0% disk.nvme1n1.out‚1397.71728515625 disk.dm-1.out‚540.5771484375 proc.cpu.threads‚17 network.recv‚154317 disk.dm-0.in‚4.3125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4.3125 network.sent‚871875) disk./.usagePercent‚76.55256253037346% disk./.usageGB‚1175.2732162475586 disk.md0.in‚0 disk.md0.out‚ 15.13671875 disk.dm-1.in‚0& disk.nvme0c0n1.out‚8505.2197265625$ memory_percent‚96.98982238769531 disk.nvme0n1.in‚4.3125 disk.nvme1c1n1.in‚0" disk.dm-0.out‚9330.43896484375 disk.rbd0.out‚0$ proc.memory.availableMB‚493.1875V`ž5ž¹‚(:“ Ÿ¸ôÒ¼ãÙÏ gpu.0.gpu‚91.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚54.0 gpu.0.powerWatts‚249.898' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚83.29933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôÌ®Ÿº‚(:” ®¸ôÒãßúË gpu.0.gpu‚66.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚54.0 gpu.0.powerWatts‚167.155' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚55.718333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Jé ‘E»‚(:º ®¸ôÒß–»Ì network.sent‚930970! disk.dm-1.out‚1031.4990234375" disk.dm-0.out‚9334.05615234375$ memory_percent‚96.99244499206543 cpu‚26.36337551142446 proc.cpu.threads‚17 network.recv‚159057) disk./.usagePercent‚76.55256004563334$ disk./.usageGB‚1175.273178100586 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚4.3125$ disk.nvme0n1.out‚8506.1005859375 disk.nvme1c1n1.in‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚4.3125& disk.nvme0c0n1.out‚8506.1005859375 disk.nvme1n1.in‚0% disk.nvme1n1.out‚1892.05126953125 disk.md0.in‚0 disk.dm-0.in‚4.3125' proc.memory.availableMB‚ 492.7578125' disk.nvme1c1n1.out‚1892.05126953125 disk.md0.out‚ 15.44921875# proc.memory.rssMB‚ 2837.51171875) proc.memory.percent‚17.31879711151123Cƒy0¼+ _wandbruntime‚441  _runtime‚441T…ã©Ã½½ & lr-AdamW/pg1‚0.0002128484733807183 & lr-AdamW/pg2‚0.0002128484733807183 trainer/global_step‚849 $ _timestamp‚1.784486965120534e+09 _runtime‚ 441.697436722 _step‚65¯&h”ǾÁ & lr-AdamW/pg2‚0.0002128484733807183 trainer/global_step‚849 $ _timestamp‚1.784486965120534e+09 _runtime‚ 441.697436722 _step‚65 & lr-AdamW/pg1‚0.0002128484733807183AìÂ0¿+ _wandbruntime‚441  _runtime‚441zrÆn»'ÀµO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8595876 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4506402 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚52513379706880 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚65029999883776 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚65029982365184 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6621848 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚51671672776192 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6621848 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538869760 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚828 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 815641778688 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚484 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 841706930688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001079 $ _timestamp‚1.784486965121189e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6759748 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚65161279035015 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚65845982447616 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15217724 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923314 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚65845624143872 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2253346 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚64347104292400 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962674 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 814514665929 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358303744 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65845624143872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785152 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15207581 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214493696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8585735 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚51671648351744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358303744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 814174742615 _step‚66 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6759477 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149207040 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 trainer/global_step‚849 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚65845982447616 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚52513230499840 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚64347121331760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8585735 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚65029999883776 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚65029982365184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 841582148096 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8585735 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚65029999883776 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚65029982365184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 841582148096 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65845624143872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785152 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15207581 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214493696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8585735 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚51671648351744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358303744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 814174742615 _step‚66 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6759477 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149207040 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 trainer/global_step‚849 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141BŽùiõ0Â+ _wandbruntime‚442  _runtime‚442¤<®5Æ&ÃÀM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚65104191652864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792320 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 815102965851 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 trainer/global_step‚849 _runtime‚ 442.052468283 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚65104174134272 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8605765 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚51730549108736 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚65921104135168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6629350 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792320 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 815442889277 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15224958 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚51730524684288 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6767014 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001079 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚67 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15235115 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚65235633340731 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚64420547414240 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214493696 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6629350 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 842675004416 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 816571689984 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚828 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 816571689984 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6629348 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923426 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15235115 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚484 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15224958 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚51730524684288 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6767014 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001079 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚67 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15235115 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚65235633340731 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚64420547414240 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214493696 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6629350 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 842675004416 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 816571689984 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚828 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 816571689984 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6629348 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923426 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15235115 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚484 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚65104191652864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792320 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 815102965851 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 trainer/global_step‚849 _runtime‚ 442.052468283 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚65104174134272 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8605765 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚51730549108736 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚65921104135168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6629350 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792320 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675CôÏw00Å+ _wandbruntime‚442  _runtime‚442ýã´ëÆû 1loss/FBt1/assignment_loss‚0.08734668946349687 .loss/FBt2/detection_loss‚5.375128671526909 % _timestamp‚1.7844869654782705e+09 _step‚68 /loss/FRt2/assignment_loss‚4.463170214804678 .loss/FRt1/detection_loss‚1.216761169859847 /loss/FRt2/detection_loss‚0.3919733523571929 1loss/SRqqt1/detection_loss‚0.9573836182978224 /loss/FBt1/detection_loss‚2.7262874836482704 &loss/total_loss‚23.562990289159977 trainer/global_step‚849 1loss/SRqqt2/detection_loss‚2.5633536693309162 epoch‚1 2loss/SRqqt1/assignment_loss‚1.1756948515451486 2loss/SRqqt2/assignment_loss‚1.3703521750028094 0loss/FBt2/assignment_loss‚0.6401508542476222 _runtime‚ 442.052931124 0loss/FRt1/assignment_loss‚2.5953875390752668\"2¦Çÿ 1loss/SRqqt2/detection_loss‚2.5633536693309162 epoch‚1 2loss/SRqqt1/assignment_loss‚1.1756948515451486 2loss/SRqqt2/assignment_loss‚1.3703521750028094 0loss/FBt2/assignment_loss‚0.6401508542476222 _runtime‚ 442.052931124 0loss/FRt1/assignment_loss‚2.5953875390752668 1loss/FBt1/assignment_loss‚0.08734668946349687 .loss/FBt2/detection_loss‚5.375128671526909 % _timestamp‚1.7844869654782705e+09 _step‚68 /loss/FRt2/assignment_loss‚4.463170214804678 .loss/FRt1/detection_loss‚1.216761169859847 /loss/FRt2/detection_loss‚0.3919733523571929 1loss/SRqqt1/detection_loss‚0.9573836182978224 /loss/FBt1/detection_loss‚2.7262874836482704 &loss/total_loss‚23.562990289159977 trainer/global_step‚849D–Ø߸FÈ‚(:» ½¸ôÒò€ëË disk.nvme0c0n1.in‚4.3125' disk.nvme0c0n1.out‚8513.78076171875 disk.nvme0n1.in‚4.3125 disk.dm-0.in‚4.3125 network.sent‚947131 network.recv‚163000% disk.nvme0n1.out‚8513.78076171875 disk.nvme1n1.in‚0% disk.nvme1n1.out‚1892.90185546875! proc.memory.rssMB‚ 2837.515625 proc.cpu.threads‚17) disk./.usagePercent‚76.55254165855648 disk.dm-1.in‚0 disk.md0.in‚0 disk.md0.out‚ 16.04296875( proc.memory.availableMB‚ 492.94921875 cpu‚26.35008386584087 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚1892.90185546875! disk.dm-0.out‚9341.2373046875$ memory_percent‚96.99127674102783) proc.memory.percent‚17.31882095336914% disk./.usageGB‚1175.2728958129883! disk.dm-1.out‚1031.5693359375 disk.rbd0.in‚0 þ>É‚(:’ ½¸ôÒ×ÿ Ì gpu.0.gpu‚39.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚267.247' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.08233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨û,ˆDÊ‚(:¹ ̸ôÒ…¥äË proc.cpu.threads‚17" disk.dm-1.out‚1031.72216796875 disk.rbd0.in‚0 network.sent‚998725 network.recv‚167794 disk.md0.in‚0 disk.md0.out‚ 16.3515625 disk.dm-0.in‚4.3125! disk.dm-0.out‚9341.8818359375) disk./.usagePercent‚76.53706147914949 disk.nvme1n1.in‚0 disk.nvme0n1.in‚4.3125$ memory_percent‚96.99270725250244! proc.memory.rssMB‚ 2837.515625) proc.memory.percent‚17.31882095336914 disk.rbd0.out‚0% disk./.usageGB‚1175.0352363586426% disk.nvme1n1.out‚1893.50146484375 disk.nvme0c0n1.in‚4.3125& disk.nvme0c0n1.out‚8514.6455078125 disk.nvme1c1n1.in‚0 disk.dm-1.in‚0$ disk.nvme0n1.out‚8514.6455078125' disk.nvme1c1n1.out‚1893.50146484375( proc.memory.availableMB‚ 492.71484375 cpu‚26.33233856383181ºeIȞ˂(:“ ̸ôÒ»—Ì gpu.0.gpu‚91.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚265.057' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.35233333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ý™†ˆ0Ì+ _wandbruntime‚465  _runtime‚465³Õ¯øÆÍÀ _runtime‚ 465.219700033 _step‚69 ' lr-AdamW/pg1‚0.00021284598466779408 ' lr-AdamW/pg2‚0.00021284598466779408 trainer/global_step‚899 % _timestamp‚1.7844869886424983e+09„— cÊÎÄ ' lr-AdamW/pg1‚0.00021284598466779408 ' lr-AdamW/pg2‚0.00021284598466779408 trainer/global_step‚899 % _timestamp‚1.7844869886424983e+09 _runtime‚ 465.219700033 _step‚69Er„ 30Ï+ _wandbruntime‚465  _runtime‚465ǸògеO K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6996946 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923314 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214493696 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2378664 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚68879383186697 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358303744 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 862169600512 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚899 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 860617807623 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 862510385664 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4760450 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2378935 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚68879026224023 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6996948 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6996946 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 $ _timestamp‚1.784486988643074e+09 _step‚70 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785152 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7139387 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 889711400960 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785152 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9091478 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4760452 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚69601740420096 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚68739588338176 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚69602098723840 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚69601740420096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚68018425455760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 860957730937 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9091478 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16078283 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6996948 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚484 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55505045305856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54615464978944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚69602098723840 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55505200804352 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155498496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131074048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚68018408416400 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚54615489403392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358303744 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0€É<¿'ѹO TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131074048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚68018408416400 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚54615489403392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358303744 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6996946 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923314 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214493696 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2378664 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚68879383186697 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358303744 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 862169600512 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚899 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 860617807623 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 862510385664 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4760450 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2378935 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚68879026224023 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6996948 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6996946 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 $ _timestamp‚1.784486988643074e+09 _step‚70 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785152 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7139387 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 889711400960 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785152 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9091478 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4760452 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚69601740420096 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚68739588338176 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚69602098723840 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚69601740420096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚68018425455760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 860957730937 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9091478 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16078283 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6996948 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚484 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55505045305856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54615464978944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚69602098723840 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55505200804352 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155498496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675F}Zîø0Ò+ _wandbruntime‚465  _runtime‚465Y¢.Å&Ó¿M MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9101367 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792320 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 890663262208 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 _step‚71 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7004450 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 863440304128 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7146911 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚69677220411392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962786 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚68091851538240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16105817 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚54674365735936 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 861546030859 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16095660 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚68953380529739 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚69676862100480 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2381124 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚484 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9091212 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7146655 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55565028998144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 890544778240 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚68953737492525 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7004448 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9101367 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4765531 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 142908416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 118483968BflÕÔÃM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16095660 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚68953380529739 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚69676862100480 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2381124 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚484 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9091212 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7146655 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55565028998144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 890544778240 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚68953737492525 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7004448 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9101367 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4765531 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 142908416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 118483968 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9101367 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792320 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 890663262208 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 _step‚71 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7004450 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 863440304128 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7146911 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚69677220411392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962786 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚68091851538240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16105817 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚54674365735936 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 861546030859 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259Glð=0Õ+ _wandbruntime‚465  _runtime‚465Ï”,Öû /loss/FRt2/detection_loss‚0.4546841125452268 0loss/FRt1/assignment_loss‚2.5430886732207405 1loss/FBt2/assignment_loss‚0.19877124529771012 1loss/SRqqt2/detection_loss‚1.9633097191531568 .loss/FBt2/detection_loss‚3.693355989197026 &loss/total_loss‚21.343747123328438 _runtime‚ 465.574332756 2loss/SRqqt2/assignment_loss‚1.3041829112200782 .loss/FBt1/detection_loss‚3.670486151919091 epoch‚1 trainer/global_step‚899 % _timestamp‚1.7844869889996784e+09 _step‚72 .loss/FRt2/assignment_loss‚4.24951075454811 1loss/SRqqt1/detection_loss‚0.7385833123253481 2loss/SRqqt1/assignment_loss‚1.0926724995407984 1loss/FBt1/assignment_loss‚0.11231428115322614 /loss/FRt1/detection_loss‚1.3227874732079246òÄwo×ÿ .loss/FRt2/assignment_loss‚4.24951075454811 1loss/SRqqt1/detection_loss‚0.7385833123253481 2loss/SRqqt1/assignment_loss‚1.0926724995407984 1loss/FBt1/assignment_loss‚0.11231428115322614 /loss/FRt1/detection_loss‚1.3227874732079246 /loss/FRt2/detection_loss‚0.4546841125452268 0loss/FRt1/assignment_loss‚2.5430886732207405 1loss/FBt2/assignment_loss‚0.19877124529771012 1loss/SRqqt2/detection_loss‚1.9633097191531568 .loss/FBt2/detection_loss‚3.693355989197026 &loss/total_loss‚21.343747123328438 _runtime‚ 465.574332756 2loss/SRqqt2/assignment_loss‚1.3041829112200782 .loss/FBt1/detection_loss‚3.670486151919091 epoch‚1 trainer/global_step‚899 % _timestamp‚1.7844869889996784e+09 _step‚72H‹K0/KØ‚(:À Û¸ôÒâõïË$ disk./.usageGB‚1175.033046722412 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚1894.14208984375 disk.nvme1n1.in‚0' disk.nvme0c0n1.out‚8520.67919921875 disk.md0.out‚ 16.66796875" disk.dm-0.out‚9347.77490234375% proc.memory.availableMB‚ 492.28125! proc.memory.rssMB‚ 2837.515625) proc.memory.percent‚17.31882095336914 cpu‚26.314642855184324 proc.cpu.threads‚17% disk.nvme1n1.out‚1894.14208984375$ memory_percent‚96.99535369873047) disk./.usagePercent‚76.53691885506674 disk.md0.in‚0 disk.dm-0.in‚4.3125" disk.dm-1.out‚1031.81591796875 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.3125 disk.nvme0n1.in‚4.3125% disk.nvme0n1.out‚8520.67919921875 network.sent‚ 1.002043e+06 network.recv‚171390<Ü5ožÙ‚(:“ Û¸ôÒÖ‚ÔÌ gpu.0.gpu‚69.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚118.622' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.54066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼‡%Ë0Ú+ _wandbruntime‚488  _runtime‚488Ëô¤ÆÛÀ _runtime‚ 488.996450126 _step‚73 ' lr-AdamW/pg1‚0.00021284179140085733 ' lr-AdamW/pg2‚0.00021284179140085733 trainer/global_step‚949 % _timestamp‚1.7844870124193695e+09ÌVèÊÜÄ ' lr-AdamW/pg2‚0.00021284179140085733 trainer/global_step‚949 % _timestamp‚1.7844870124193695e+09 _runtime‚ 488.996450126 _step‚73 ' lr-AdamW/pg1‚0.00021284179140085733IƱ;0Ý+ _wandbruntime‚488  _runtime‚488´æ ¼'Þ¶O HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚72449176792576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚73358215000064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358303744 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 908697422336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785152 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16959128 % _timestamp‚1.7844870124199822e+09 _step‚74 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 trainer/global_step‚949 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚828 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚72449176792576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚71689729579760 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚72449159273984 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7372048 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9576939 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58496988483584 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚71689712540400 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16948985 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214493696 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚57559306030592 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9587080 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2504011 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214493696 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7518800 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7372048 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚73357856696320 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚72449159273984 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7372046 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 937557670400 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962674 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446¬ýÀ'ߺO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785152 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16959128 % _timestamp‚1.7844870124199822e+09 _step‚74 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 trainer/global_step‚949 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚828 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚72449176792576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚71689729579760 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚72449159273984 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7372048 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9576939 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58496988483584 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚71689712540400 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16948985 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214493696 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚57559306030592 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9587080 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2504011 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214493696 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7518800 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7372048 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚73357856696320 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚72449159273984 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7372046 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 937557670400 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962674 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚72449176792576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚73358215000064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358303744 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 908697422336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538869760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72523368561664 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7379548 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58556828376064 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16966362 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 938508654592 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001079 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7526111 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 909968125952 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16976519 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7379550 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9596969 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚57618157938688 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 938646012928 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161782784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16966362 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2506530 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚72671484681533 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923426 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16976519 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 909627333632 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚72671127718747 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚73433336687616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚57618182363136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚949 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58556666593280 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7379550 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7526400 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358310912 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2506817 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9586814 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538869760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72523368561664 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7379548 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58556828376064 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16966362 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 938508654592 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001079 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7526111 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 909968125952 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16976519 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7379550 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9596969 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚57618157938688 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 938646012928 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161782784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16966362 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2506530 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚72671484681533 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923426 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16976519 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 909627333632 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚72671127718747 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚73433336687616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚57618182363136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚949 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58556666593280 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7379550 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7526400 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358310912 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2506817 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9586814 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10082682 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚75361016664400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17819687 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 955225244160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10072541 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7747146 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2629614 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10072541 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923314 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 985502196736 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5268550 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚61488600430080 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241367 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚76158765246976 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚484 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358303744 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚76158747728384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131074048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚999 _runtime‚ 512.790369913 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 955566029312 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7747148 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214493696 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 955225244160 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 955566029312 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155498496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214493696 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 _step‚78 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7747148 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17829830 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7898166 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚77113972972544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358303744%5ñÎ@ï½O JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 _step‚78 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7747148 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17829830 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7898166 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚77113972972544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358303744 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚60503122657792 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 953503937639 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚76158765246976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001079 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚77114331276288 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7747146 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚828 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10082682 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚75361016664400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17819687 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 955225244160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10072541 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7747146 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2629614 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10072541 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923314 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 985502196736 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5268550 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚61488600430080 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241367 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚76158765246976 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚484 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358303744 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚76158747728384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131074048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚999 _runtime‚ 512.790369913 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 955566029312 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7747148 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214493696 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 955225244160 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 955566029312 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155498496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214493696 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0NFiõ,0ð+ _wandbruntime‚513  _runtime‚513ɦÎùÉ&ñÃM I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7754648 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792320 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚60561974565888 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚61548436422144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 954432160875 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7844870365708017e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5273631 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923426 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10092571 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 956155155456 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 956495947776 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17847221 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 986597117440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135261184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚76232957016064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚60561998990336 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962786 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75434442746880 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214493696 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7905378 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 986461856256 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚76389231870541 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚77189094652928 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159685632 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚76232957016064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7754650 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚828 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚76388874907755 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚77189452963840 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2631747 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚77189452963840 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10082416 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚999 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17837064 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚61548596107776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001079 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5273633 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10092571 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 956155155456 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 956495947776 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17847221 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 986597117440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135261184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚76232957016064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚60561998990336 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962786 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75434442746880 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214493696 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7905378 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 986461856256 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚76389231870541 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚77189094652928 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159685632 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚76232957016064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7754650 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚828 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚76388874907755 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚77189452963840 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2631747 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚77189452963840 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10082416 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚999 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17837064 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚61548596107776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001079 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5273633 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 954432160875 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0Oét~—0ó+ _wandbruntime‚513  _runtime‚513{nm~ôû 2loss/SRqqt1/assignment_loss‚0.9728077739767557 2loss/SRqqt2/assignment_loss‚1.5902207616836794 /loss/FRt1/detection_loss‚1.2357309480488092 /loss/FRt2/detection_loss‚0.5534827666754789 0loss/SRqqt2/detection_loss‚2.012497262836945 .loss/FBt1/detection_loss‚7.314814694464503 /loss/FBt2/detection_loss‚1.8792471313476562 _runtime‚ 513.146371051 /loss/FRt1/assignment_loss‚2.495908263987965 1loss/FBt1/assignment_loss‚0.14358600721172418 trainer/global_step‚999 _step‚80 0loss/FRt2/assignment_loss‚4.1554552231948385 &loss/total_loss‚23.550978550274728 epoch‚1 $ _timestamp‚1.784487036571712e+09 1loss/FBt2/assignment_loss‚0.42302598216570914 1loss/SRqqt1/detection_loss‚0.7742017346806642ß¿HÜõÿ .loss/FBt1/detection_loss‚7.314814694464503 /loss/FBt2/detection_loss‚1.8792471313476562 _runtime‚ 513.146371051 /loss/FRt1/assignment_loss‚2.495908263987965 1loss/FBt1/assignment_loss‚0.14358600721172418 trainer/global_step‚999 _step‚80 0loss/FRt2/assignment_loss‚4.1554552231948385 &loss/total_loss‚23.550978550274728 epoch‚1 $ _timestamp‚1.784487036571712e+09 1loss/FBt2/assignment_loss‚0.42302598216570914 1loss/SRqqt1/detection_loss‚0.7742017346806642 2loss/SRqqt1/assignment_loss‚0.9728077739767557 2loss/SRqqt2/assignment_loss‚1.5902207616836794 /loss/FRt1/detection_loss‚1.2357309480488092 /loss/FRt2/detection_loss‚0.5534827666754789 0loss/SRqqt2/detection_loss‚2.012497262836945P‚5BJö‚(:¿ ˆ¹ôÒŠ‡•Ì disk.nvme1n1.in‚0$ memory_percent‚96.99623584747314( proc.memory.availableMB‚ 492.13671875) proc.memory.percent‚17.31882095336914 cpu‚26.269783627809964 network.recv‚184630" disk.dm-1.out‚1032.30126953125 disk.nvme0c0n1.in‚4.3125' disk.nvme0c0n1.out‚8746.15869140625& disk.nvme1c1n1.out‚1927.3935546875$ disk.nvme1n1.out‚1927.3935546875 disk.md0.out‚ 18.12109375) disk./.usagePercent‚76.55245295333427 disk.dm-0.in‚4.3125 disk.dm-0.out‚ 9602.91796875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚4.3125% disk./.usageGB‚1175.2715339660645 disk.dm-1.in‚0% disk.nvme0n1.out‚8746.15869140625 disk.md0.in‚0! proc.memory.rssMB‚ 2837.515625 proc.cpu.threads‚17 network.sent‚ 1.145398e+06 disk.nvme1c1n1.in‚0¸êoÛ“÷‚(:ˆ ˆ¹ôÒÔÈÓÌ gpu.0.gpu‚59.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚226.464' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚75.488 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×SÇg0ø+ _wandbruntime‚536  _runtime‚536Ì4èpÅù¿ & lr-AdamW/pg1‚0.0002128282915009017 & lr-AdamW/pg2‚0.0002128282915009017 trainer/global_step‚1049 % _timestamp‚1.7844870600854142e+09 _runtime‚ 536.661418275 _step‚81•rñGÉúà trainer/global_step‚1049 % _timestamp‚1.7844870600854142e+09 _runtime‚ 536.661418275 _step‚81 & lr-AdamW/pg1‚0.0002128282915009017 & lr-AdamW/pg2‚0.0002128282915009017QxNLÜ0û+ _wandbruntime‚536  _runtime‚536xSíTüÂO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10578284 ]ODeviceStatsMonitor.on_t'Ûˆƒt&rain_batch_start/inactive_split_bytes.small_pool.current‚ 124782592 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10568143 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64480552290816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚79868336182784 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10578284 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923314 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1001753065984 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18700532 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚79868353701376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8276757 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241367 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚79032337827760 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358303744 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785152 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚80870089248768 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8122246 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8277029 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538869760 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2754427 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962674 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001079 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚79868336182784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358303744 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚79032320788400 _step‚82 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚80870447552512 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538869760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 trainer/global_step‚1049 _runtime‚ 536.661476686 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214493696 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149207040 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚64480403083776 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1033488223232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1002093851136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚79868353701376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5522602 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1001753065984 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8122248 ‰3ÉÌ'ýÆO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1001753065984 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8122248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10578284 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124782592 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10568143 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64480552290816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚79868336182784 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10578284 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923314 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1001753065984 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18700532 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚79868353701376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8276757 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241367 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚79032337827760 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358303744 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785152 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚80870089248768 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8122246 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8277029 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538869760 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2754427 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962674 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001079 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚79868336182784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358303744 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚79032320788400 _step‚82 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚80870447552512 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538869760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 trainer/global_step‚1049 _runtime‚ 536.661476686 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214493696 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149207040 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚64480403083776 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1033488223232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1002093851136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚79868353701376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5522602 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0RÞEk0þ+ _wandbruntime‚537  _runtime‚537m¸]Ó&ÿÍM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18707766 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8129750 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8284540 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8284257 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001079 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792320 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚484 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚79105746870880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1000875225883 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚64540241918464 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10578018 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚828 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚79942527951872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚79942545470464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64540393215488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚1049 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18717923 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚79942527951872 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214493696 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚80945569240064 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 _step‚83 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1034577597952 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538869760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2756857 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1001215149309 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1002682977280 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8129748 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5527681 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80106979059549 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1003023769600 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚80945210929152 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1003023769600 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2756576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923426 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18707766 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80106622096763 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792320 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8129750 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18717923 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8284257 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001079 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792320 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚484 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚79105746870880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1000875225883 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚64540241918464 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10578018 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚828 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚79942527951872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.all­°=ocated‚79942545470464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64540393215488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚1049 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18717923 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚79942527951872 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214493696 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚80945569240064 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 _step‚83 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1034577597952 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538869760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2756857 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1001215149309 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1002682977280 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8129748 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5527681 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80106979059549 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1003023769600 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚80945210929152 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1003023769600 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2756576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923426 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18707766 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80106622096763 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792320 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8129750 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18717923 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 ODeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚1099 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11063745 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19561091 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8497346 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚83577942155776 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1048280887808 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚84626563828736 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚66390755912192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1081637959168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚83577924637184 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚84626563828736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11063745 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚830 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001079 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19571234 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8497346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153401344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538869760 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11073886 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8497348 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84626205524992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚83577942155776 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128976896 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5776650 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚84626205524992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358303744 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1048621672960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1048621672960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚82703624912400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1046390067655 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 _step‚86 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8656110 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664GѽÍ'ÇO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214493696 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8497348 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚1099 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11063745 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19561091 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8497346 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚83577942155776 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1048280887808 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚84626563828736 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚66390755912192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1081637959168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚83577924637184 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚84626563828736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11063745 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚830 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001079 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19571234 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8497346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153401344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538869760 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11073886 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8497348 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84626205524992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚83577942155776 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128976896 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5776650 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚84626205524992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358303744 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1048621672960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1048621672960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚82703624912400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1046390067655 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 _step‚86 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8656110 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67472393871360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚83750014980055 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82703641951760 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚83750371942729 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2879460 _runtime‚ 560.168841302 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚83577924637184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1048280887808 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8656404 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7844870835939767e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358303744 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚66390731487744 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.a¹ÿJ‚#%llocated‚ 1082595158016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11073620 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8663721 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001079 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84701685516288 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962786 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1047318290891 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19578468 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚83824726248557 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚82777068034240 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19588625 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1049551591424 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2881990 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚83652133924864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214493696 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8504850 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5781733 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11073620 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚83824369285771 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5781731 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8504850 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1049551591424 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67532227402752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1082468285440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151297024 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚83652133924864 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1049210799104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126872576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11083775 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚84701327205376 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1047658214317 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19578468 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19588625 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358310912 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚830 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67532076105728 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚66449632244736 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0‡~×&ÑM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚83652133924864 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1049210799104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126872576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11083775 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚84701327205376 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1047658214317 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19578468 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19588625 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358310912 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚830 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67532076105728 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚66449632244736 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8504848 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚82777050994880 _step‚87 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1082595158016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11073620 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8663721 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001079 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84701685516288 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962786 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1047318290891 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19578468 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚83824726248557 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚82777068034240 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19588625 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1049551591424 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2881990 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚83652133924864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214493696 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8504850 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5781733 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11073620 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚83824369285771 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5781731 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8504850 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1049551591424 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67532227402752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1082468285440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151297024 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372W‰èÂh0‘+ _wandbruntime‚560  _runtime‚560,çþ’ø &loss/total_loss‚17.484532637711638 % _timestamp‚1.7844870839511669e+09 /loss/FRt1/assignment_loss‚2.107125556002782 1loss/SRqqt1/detection_loss‚0.6978968355724131 /loss/FBt2/detection_loss‚2.2033226084709168 trainer/global_step‚1099 _runtime‚ 560.525914483 /loss/FRt1/detection_loss‚1.2726133455242143 _step‚88 0loss/SRqqt2/assignment_loss‚1.46533951624904 1loss/FBt1/assignment_loss‚0.16036751852101286 epoch‚1 .loss/FRt2/assignment_loss‚4.14623555253605 1loss/SRqqt1/assignment_loss‚0.920048609902624 1loss/FBt2/assignment_loss‚0.38752409379929303 .loss/FRt2/detection_loss‚0.597423593449255 0loss/SRqqt2/detection_loss‚1.744047500702647 /loss/FBt1/detection_loss‚1.7825879069813906žæ… “ü .loss/FRt2/assignment_loss‚4.14623555253605 1loss/SRqqt1/assignment_loss‚0.920048609902624 1loss/FBt2/assignment_loss‚0.38752409379929303 .loss/FRt2/detection_loss‚0.597423593449255 0loss/SRqqt2/detection_loss‚1.744047500702647 /loss/FBt1/detection_loss‚1.7825879069813906 &loss/total_loss‚17.484532637711638 % _timestamp‚1.7844870839511669e+09 /loss/FRt1/assignment_loss‚2.107125556002782 1loss/SRqqt1/detection_loss‚0.6978968355724131 /loss/FBt2/detection_loss‚2.2033226084709168 trainer/global_step‚1099 _runtime‚ 560.525914483 /loss/FRt1/detection_loss‚1.2726133455242143 _step‚88 0loss/SRqqt2/assignment_loss‚1.46533951624904 1loss/FBt1/assignment_loss‚0.16036751852101286 epoch‚1Xu]ÑH”‚(:½ µ¹ôÒÉ£ÌË network.recv‚197143 disk.rbd0.in‚0 disk.md0.in‚0 disk.md0.out‚ 19.109375" disk.dm-0.out‚9624.16943359375 cpu‚26.231063498994974 network.sent‚ 1.264152e+06) disk./.usagePercent‚76.54971626056869 disk.nvme0c0n1.in‚4.3125 disk.dm-0.in‚4.3125! proc.memory.rssMB‚ 2837.515625) proc.memory.percent‚17.31882095336914 proc.cpu.threads‚17% disk./.usageGB‚1175.2295188903809 disk.nvme0n1.in‚4.3125 disk.nvme1n1.in‚0 disk.dm-1.in‚0' proc.memory.availableMB‚ 491.6171875 disk.rbd0.out‚0& disk.nvme0c0n1.out‚8765.9267578125$ disk.nvme0n1.out‚8765.9267578125 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚1931.32666015625% disk.nvme1n1.out‚1931.32666015625! disk.dm-1.out‚1032.6083984375# memory_percent‚96.9994068145752c(Éùž•‚(:“ µ¹ôÒî÷íË gpu.0.gpu‚72.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚265.806' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.60199999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 oø0–+ _wandbruntime‚583  _runtime‚583ˆÆ—À ' lr-AdamW/pg2‚0.00021280797454576114 trainer/global_step‚1149 $ _timestamp‚1.784487107401852e+09 _runtime‚ 583.976817164 _step‚89 ' lr-AdamW/pg1‚0.00021280797454576114ži‘ŽÊ˜Ä ' lr-AdamW/pg1‚0.00021280797454576114 ' lr-AdamW/pg2‚0.00021280797454576114 trainer/global_step‚1149 $ _timestamp‚1.784487107401852e+09 _runtime‚ 583.976817164 _step‚89YÙÌ(0™+ _wandbruntime‚583  _runtime‚583É<€C¬&šÃO \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚87287530610176 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚87287530610176 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358303744 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1094808709632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128976896 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9036101 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚69334548114944 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚87468119131737 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1092833132663 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20441936 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚70464177894400 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚86374929036400 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214493696 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923314 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚88382321801216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1093173055977 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1095149494784 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8872446 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11559347 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88382321801216 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358303744 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11559347 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚69334572539392 _step‚90 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚88382680104960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8872448 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚87467762169063 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1094808709632 trainer/global_step‚1149 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214493696 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20441936 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚70464177894400 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚86374929036400 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214493696 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923314 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚88382321801216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1093173055977 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1095149494784 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8872446 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11559347 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88382321801216 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358303744 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11559347 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚69334572539392 _step‚90 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚88382680104960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8872448 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚87467762169063 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1094808709632 trainer/global_step‚1149 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214493696 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358303744 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1094808709632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128976896 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259Z0zë0œ+ _wandbruntime‚584  _runtime‚584<¥7Ó&ÍM QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8879950 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚70523854566400 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚87542116474779 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20449170 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11579377 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚87361704860672 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1130565380096 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135261184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962786 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚86448355118880 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1093761355899 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001079 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1094101279325 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 _step‚91 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11569222 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358310912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20459327 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9043442 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚70524014252032 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚88457443481600 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3007946 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚88457801792512 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538869760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20459327 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20449170 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6035781 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923426 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1096079413248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1130430118912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚1149 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159685632 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8879948 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9043729 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1095738620928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1096079413248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚88457801792512 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792320 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11569222 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚87361722379264 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 % _timestamp‚1.7844871077587602e+09 _runtime‚ 584.333882534©6C7×&žÑM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135261184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962786 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚86448355118880 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1093761355899 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001079 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1094101279325 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 _step‚91 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11569222 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358310912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20459327 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9043442 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚70524014252032 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚88457443481600 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3007946 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚88457801792512 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538869760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20459327 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20449170 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6035781 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923426 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1096079413248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1130430118912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚1149 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159685632 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8879948 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9043729 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1095738620928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1096079413248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚88457801792512 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792320 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11569222 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚87361722379264 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 % _timestamp‚1.7844871077587602e+09 _runtime‚ 584.333882534 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8879950 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚70523854566400 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚87542116474779 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20449170 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11579377 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚87361704860672 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1130565380096[.-ñP0Ÿ+ _wandbruntime‚584  _runtime‚584r!‘ÿÿ ù /loss/FRt2/detection_loss‚0.6367540971554254 epoch‚1 /loss/FRt2/assignment_loss‚4.052941237304573 1loss/FBt1/assignment_loss‚0.09311362916373406 0loss/SRqqt1/detection_loss‚0.605317616405632 ,loss/FBt1/detection_loss‚2.1633008251538 1loss/SRqqt2/detection_loss‚1.7419766899639741 /loss/FBt2/detection_loss‚2.4886142276227474 &loss/total_loss‚17.821484835333727 0loss/FRt1/assignment_loss‚1.9895466631141032 2loss/SRqqt2/assignment_loss‚1.3598709536201663 trainer/global_step‚1149 % _timestamp‚1.7844871077597876e+09 _runtime‚ 584.334447355 _step‚92 2loss/SRqqt1/assignment_loss‚0.9373127720330464 0loss/FBt2/assignment_loss‚0.3999178989251959 .loss/FRt1/detection_loss‚1.352818224871328O¡ý /loss/FRt2/assignment_loss‚4.052941237304573 1loss/FBt1/assignment_loss‚0.09311362916373406 0loss/SRqqt1/detection_loss‚0.605317616405632 ,loss/FBt1/detection_loss‚2.1633008251538 1loss/SRqqt2/detection_loss‚1.7419766899639741 /loss/FBt2/detection_loss‚2.4886142276227474 &loss/total_loss‚17.821484835333727 0loss/FRt1/assignment_loss‚1.9895466631141032 2loss/SRqqt2/assignment_loss‚1.3598709536201663 trainer/global_step‚1149 % _timestamp‚1.7844871077597876e+09 _runtime‚ 584.334447355 _step‚92 2loss/SRqqt1/assignment_loss‚0.9373127720330464 0loss/FBt2/assignment_loss‚0.3999178989251959 .loss/FRt1/detection_loss‚1.352818224871328 /loss/FRt2/detection_loss‚0.6367540971554254 epoch‚1\ÝZ¸ž¢‚(:“ ĹôÒåÀÅË gpu.0.gpu‚75.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚119.141' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.71366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–Í„Î&£‚(:¿ ĹôÒ³ˆÌ! disk.dm-1.out‚1032.6943359375( proc.memory.availableMB‚ 491.32421875) proc.memory.percent‚17.31882095336914 proc.cpu.threads‚17 network.recv‚201789) disk./.usagePercent‚76.54971551514666$ disk./.usageGB‚1175.229507446289 disk.md0.in‚ªÄ$0 disk.md0.out‚ 19.43359375 disk.dm-1.in‚0& disk.nvme0c0n1.out‚8767.0341796875 disk.nvme1c1n1.in‚0 network.sent‚ 1.338462e+06" disk.dm-0.out‚9624.97802734375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.3125$ disk.nvme0n1.out‚8767.0341796875' disk.nvme1c1n1.out‚1931.82080078125$ memory_percent‚97.00119495391846 disk.dm-0.in‚4.3125 disk.nvme0n1.in‚4.3125! proc.memory.rssMB‚ 2837.515625 cpu‚26.21939459813904 disk.nvme1n1.in‚0% disk.nvme1n1.out‚1931.820800781254§•“E¤‚(:º Ó¹ôÒˆË°Ë proc.cpu.threads‚17 network.recv‚207301) disk./.usagePercent‚76.54971775141276 disk.nvme0n1.in‚4.3125 disk.md0.out‚ 19.80859375" disk.dm-0.out‚9628.12060546875 disk.rbd0.out‚0& proc.memory.availableMB‚ 491.234375! proc.memory.rssMB‚ 2837.515625 disk.nvme0c0n1.in‚4.3125" disk.nvme0n1.out‚ 8770.27734375& disk.nvme1c1n1.out‚1935.8837890625$ disk.nvme1n1.out‚1935.8837890625 disk.dm-0.in‚4.3125 disk.rbd0.in‚0$ memory_percent‚97.00174331665039) proc.memory.percent‚17.31882095336914 disk.nvme1n1.in‚0 disk.dm-1.in‚0 cpu‚26.209203483008487 disk.md0.in‚0 network.sent‚ 1.388874e+06% disk./.usageGB‚1175.2295417785645$ disk.nvme0c0n1.out‚ 8770.27734375 disk.nvme1c1n1.in‚0" disk.dm-1.out‚1036.03955078125ÑV4Už¥‚(:“ Ó¹ôÒôáøË gpu.0.gpu‚71.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚263.227' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.74233333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í=ɱ0¦+ _wandbruntime‚607  _runtime‚607}JŠéħ¾ trainer/global_step‚1199 $ _timestamp‚1.784487131300025e+09 _runtime‚ 607.875145843 _step‚93 & lr-AdamW/pg1‚0.0002127952600622079 & lr-AdamW/pg2‚0.0002127952600622079í—|åȨ _step‚93 & lr-AdamW/pg1‚0.0002127952600622079 & lr-AdamW/pg2‚0.0002127952600622079 trainer/global_step‚1199 $ _timestamp‚1.784487131300025e+09 _runtime‚ 607.875145843]\ý‰0©+ _wandbruntime‚607  _runtime‚607½OvÈ'ªÂO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1139276197671 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358303744 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12065090 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚91185866320745 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1141677316608 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚830 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6284752 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9247548 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962674 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 _runtime‚ 607.876359505 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214493696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1177527201280 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12054949 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1139616120985 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785152 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚92138438077440 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚90997101545984 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3131880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1141336531456 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21302495 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚91185509358071 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚92138796381184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358303744 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1177396127232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90046233160400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚90997119064576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚73455760869376 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚72278364742144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚92138796381184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131074048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚90046250199760 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1141677316608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923314 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 trainer/global_step‚1199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92138438077440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21312638 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6284750 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241367 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1141336531456 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280.ƒ»cÌ'«ÆO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21302495 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚91185509358071 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚92138796381184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358303744 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1177396127232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90046233160400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚90997119064576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚73455760869376 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚72278364742144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚92138796381184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131074048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚90046250199760 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1141677316608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923314 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 trainer/global_step‚1199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92138438077440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21312638 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6284750 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241367 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1141336531456 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1139276197671 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358303744 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12065090 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚91185866320745 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1141677316608 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚830 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6284752 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9247548 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962674 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 _runtime‚ 607.876359505 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214493696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1177527201280 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12054949 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1139616120985 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785152 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚92138438077440 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚90997101545984 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3131880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚486 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1141336531456 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0^ò÷1M0¬+ _wandbruntime‚608  _runtime‚608þP 3Ó&­ÍM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1142266442752 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9255048 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9424218 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3134385 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1140544344333 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚91071310833664 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚72337241074688 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1140204420907 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 trainer/global_step‚1199 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1142607235072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21330029 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9423945 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6289831 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157588480 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133164032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12074979 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538869760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1178356413440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1142266442752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9255050 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538869760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12064824 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962786 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚92213918068736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 _runtime‚ 608.234192388 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚91071310833664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚91071293315072 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚73515597488128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214493696 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚72337265499136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0na"®ÑM ZGDeviceStatsMonitor.on_trÃäµ&ain_batch_end/active_bytes.large_pool.allocated‚91071310833664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚91071293315072 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚73515597488128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214493696 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚72337265499136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1142266442752 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9255048 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9424218 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3134385 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1140544344333 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚91071310833664 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚72337241074688 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1140204420907 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 trainer/global_step‚1199 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1142607235072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21330029 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9423945 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6289831 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157588480 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133164032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12074979 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538869760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1178356413440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1142266442752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9255050 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538869760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12064824 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962786 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚92213918068736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 _runtime‚ 608.234192388_]êºö0¯+ _wandbruntime‚608  _runtime‚608û»J°ü trainer/global_step‚1199 _step‚96 .loss/FBt2/detection_loss‚9.843642078732916 epoch‚1 /loss/FRt2/assignment_loss‚3.712949586375324 1loss/FBt2/assignment_loss‚0.39156741279753315 1loss/SRqqt1/detection_loss‚0.6516721602653446 1loss/SRqqt2/detection_loss‚1.5363013873516944 &loss/total_loss‚24.430174308737087 % _timestamp‚1.7844871316600358e+09 _runtime‚ 608.234729129 0loss/FRt1/assignment_loss‚1.9074755559056442 2loss/SRqqt2/assignment_loss‚1.0441682542416644 .loss/FRt1/detection_loss‚1.269566623317225 .loss/FBt1/detection_loss‚2.483354208735063 2loss/SRqqt1/assignment_loss‚0.9758144664113707 1loss/FBt1/assignment_loss‚0.07064579436364328 /loss/FRt2/detection_loss‚0.5430167802396637ŸIŸá±€ 1loss/SRqqt2/detection_loss‚1.5363013873516944 &loss/total_loss‚24.430174308737087 % _timestamp‚1.7844871316600358e+09 _runtime‚ 608.234729129 0loss/FRt1/assignment_loss‚1.9074755559056442 2loss/SRqqt2/assignment_loss‚1.0441682542416644 .loss/FRt1/detection_loss‚1.269566623317225 .loss/FBt1/detection_loss‚2.483354208735063 2loss/SRqqt1/assignment_loss‚0.9758144664113707 1loss/FBt1/assignment_loss‚0.07064579436364328 /loss/FRt2/detection_loss‚0.5430167802396637 trainer/global_step‚1199 _step‚96 .loss/FBt2/detection_loss‚9.843642078732916 epoch‚1 /loss/FRt2/assignment_loss‚3.712949586375324 1loss/FBt2/assignment_loss‚0.39156741279753315 1loss/SRqqt1/detection_loss‚0.6516721602653446`%ʉD²‚(:¹ â¹ôÒ¬–˜Ì$ disk.nvme1n1.out‚1936.4814453125 network.sent‚ 1.408142e+06% disk./.usageGB‚1175.3501472473145 disk.nvme0c0n1.in‚4.3125! disk.nvme0n1.out‚ 8835.9921875 disk.nvme1n1.in‚0 disk.rbd0.in‚0# disk.nvme0c0n1.out‚ 8835.9921875 disk.dm-0.in‚4.3125" disk.dm-0.out‚9693.55029296875" disk.dm-1.out‚1036.18017578125$ memory_percent‚97.00207710266113' proc.memory.availableMB‚ 491.1796875 proc.cpu.threads‚17 disk.md0.in‚0 disk.md0.out‚ 20.1640625 network.recv‚211719) disk./.usagePercent‚76.55757350576873 disk.nvme0n1.in‚4.3125 disk.rbd0.out‚0 cpu‚26.20016783472646 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚1936.4814453125 disk.dm-1.in‚0# proc.memory.rssMB‚ 2837.51953125) proc.memory.percent‚17.31884479522705½Õgž³‚(:“ â¹ôÒ°öÌ gpu.0.gpu‚71.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚265.697' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.56566666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03²œÚž´‚(:“ ñ¹ôÒ„͉Ì gpu.0.gpu‚82.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚266.984' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.99466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ9&ôNµ‚(:à ñ¹ôÒªåÌ( disk./.usagePercent‚76.5575834447292 disk.dm-0.in‚4.3125" disk.dm-0.out‚9694.70556640625 disk.rbd0.in‚0' disk.nvme0c0n1.out‚8837.39697265625" disk.dm-1.out‚1036.26904296875 disk.rbd0.out‚0$ memory_percent‚96.99912071228027' proc.memory.availableMB‚ 491.6640625 cpu‚26.189630862926798 proc.cpu.threads‚17 disk.nvme0n1.in‚4.3125 network.sent‚ 1.457314e+06 disk.dm-1.in‚0 disk.nvme0c0n1.in‚4.3125' disk.nvme1c1n1.out‚1936.93701171875 disk.nvme1n1.in‚0) proc.memory.percent‚17.31884479522705 network.recv‚214817$ disk./.usageGB‚1175.350299835205 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚1936.93701171875 disk.md0.in‚0 disk.md0.out‚ 20.44140625# proc.memory.rssMB‚ 2837.51953125% disk.nvme0n1.out‚8837.39697265625÷$×`0¶+ _wandbruntime‚631  _runtime‚631žÏäÅ·¿ _step‚97 & lr-AdamW/pg1‚0.0002127808418370996 & lr-AdamW/pg2‚0.0002127808418370996 trainer/global_step‚1249 % _timestamp‚1.7844871551676233e+09 _runtime‚ 631.7426033714ºvɸà & lr-AdamW/pg1‚0.0002127808418370996 & lr-AdamW/pg2‚0.0002127808418370996 trainer/global_step‚1249 % _timestamp‚1.7844871551676233e+09 _runtime‚ 631.742603371 _step‚97a'EãX0¹+ _wandbruntime‚631  _runtime‚631§ ¿É'ºÃO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785152 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12550551 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚95894554353664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12550551 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214493696 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚830 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚94903613509753 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1185719262679 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241367 trainer/global_step‚1249 % _timestamp‚1.7844871551686065e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214493696 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9622646 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358303744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1225520632320 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9622648 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚95894912657408 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923314 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1187864353280 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9796424 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚95894912657408 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3257624 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001079 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1187864353280 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9796706 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9622646 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3257904 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚76447581413376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0O˜¦$,»ÇO XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚94706690000384 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22183340 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1225400044032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 _step‚98 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚95894554353664 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538869760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 _runtime‚ 631.747420649 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1188205138432 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538869760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12560692 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚94706707518976 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚94706690000384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 145012736 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9796706 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9622646 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3257904 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚76447581413376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785152 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12550551 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚95894554353664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12550551 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214493696 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚830 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚94903613509753 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1185719262679 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241367 trainer/global_step‚1249 % _timestamp‚1.7844871551686065e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214493696 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9622646 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358303744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1225520632320 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9622648 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚95894912657408 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923314 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1187864353280 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9796424 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚95894912657408 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3257624 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001079 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1187864353280 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9804032 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128969728 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923426 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962786 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 trainer/global_step‚1249 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚94780899288064 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792320 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 _step‚99 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9630150 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1226355116032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1189135056896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1189135056896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9630150 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12570581 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚95970034344960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76507566212096 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚95970034344960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358310912 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚75281057701888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214493696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792320 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22190574 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76507412817920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22190574 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538869760 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3260432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1188794264576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1226484085760 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚93790980406240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12570581 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12560426 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22200731 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1188794264576 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6543883 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12560426 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚94977967815581 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001079 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1186647485915 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22200731 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 _runtime‚ 632.100598928 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9630148 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9630148 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3260151 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚75281082126336 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9804315 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 % _timestamp‚1.7844871555254536e+09åG >×&¾ÑM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76507566212096 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚95970034344960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358310912 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚75281057701888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214493696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792320 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22190574 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76507412817920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22190574 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538869760 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3260432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1188794264576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1226484085760 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚93790980406240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12570581 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12560426 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22200731 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1188794264576 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6543883 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12560426 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚94977967815581 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001079 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1186647485915 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22200731 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 _runtime‚ 632.100598928 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9630148 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9630148 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3260151 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚75281082126336 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9804315 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 % _timestamp‚1.7844871555254536e+09 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9804032 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128969728 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923426 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962786 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 trainer/global_step‚1249 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚94780899288064 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792320 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 _step‚99 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9630150 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1226355116032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1189135056896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1189135056896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9630150 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12570581 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚95970034344960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934c+¯ÛÎ0¿+ _wandbruntime‚632  _runtime‚632kˆ#Àþ 1loss/FBt2/assignment_loss‚0.13350811536057497 epoch‚1 trainer/global_step‚1249 % _timestamp‚1.7844871555263877e+09 _runtime‚ 632.101033799 _step‚100 %loss/total_loss‚16.23157094432016 1loss/SRqqt1/assignment_loss‚0.902231660928743 2loss/SRqqt2/assignment_loss‚1.1785537484558481 /loss/FRt2/detection_loss‚0.6433135992368721 /loss/FBt2/detection_loss‚2.5964789454113033 0loss/FRt2/assignment_loss‚3.4709890406143846 1loss/FBt1/assignment_loss‚0.07888953944728208 /loss/FBt1/detection_loss‚1.8405191692586818 0loss/FRt1/assignment_loss‚1.8010388754663014 /loss/FRt1/detection_loss‚1.2968202203874983 1loss/SRqqt1/detection_loss‚0.7090896149953555 0loss/SRqqt2/detection_loss‚1.580138414757314é]ÃäÁ‚ _step‚100 %loss/total_loss‚16.23157094432016 1loss/SRqqt1/assignment_loss‚0.902231660928743 2loss/SRqqt2/assignment_loss‚1.1785537484558481 /loss/FRt2/detection_loss‚0.6433135992368721 /loss/FBt2/detection_loss‚2.5964789454113033 0loss/FRt2/assignment_loss‚3.4709890406143846 1loss/FBt1/assignment_loss‚0.07888953944728208 /loss/FBt1/detection_loss‚1.8405191692586818 0loss/FRt1/assignment_loss‚1.8010388754663014 /loss/FRt1/detection_loss‚1.2968202203874983 1loss/SRqqt1/detection_loss‚0.7090896149953555 0loss/SRqqt2/detection_loss‚1.580138414757314 1loss/FBt2/assignment_loss‚0.13350811536057497 epoch‚1 trainer/global_step‚1249 % _timestamp‚1.7844871555263877e+09 _runtime‚ 632.101033799d¦ýF‚(:» €ºôÒ€ÂëË disk.dm-1.in‚0 disk.rbd0.out‚0& disk.nvme1c1n1.out‚1937.6083984375 disk.nvme1n1.in‚0 disk.md0.out‚20.84375) disk./.usagePercent‚76.54209581110186" disk.dm-0.out‚9695.41259765625# disk.nvme0n1.out‚8838.447265625 disk.nvme1c1n1.in‚0# proc.memory.rssMB‚ 2837.51953125) proc.memory.percent‚17.31884479522705% disk.nvme0c0n1.out‚8838.447265625 network.sent‚ 1.460563e+06 network.recv‚219206" disk.dm-1.out‚1036.40185546875 disk.rbd0.in‚0$ disk.nvme1n1.out‚1937.6083984375 disk.md0.in‚0 cpu‚26.180459941615467 disk.nvme0n1.in‚4.3125$ memory_percent‚96.99945449829102& proc.memory.availableMB‚ 491.609375 proc.cpu.threads‚17 disk.nvme0c0n1.in‚4.3125% disk./.usageGB‚1175.1125259399414 disk.dm-0.in‚4.3125Û`éSžÃ‚(:“ €ºôÒö«ØÌ gpu.0.gpu‚92.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚56.0 gpu.0.powerWatts‚256.339' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.44633333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00.DÉ0Ä+ _wandbruntime‚654  _runtime‚654!„eìÅÅ¿ _step‚101 & lr-AdamW/pg1‚0.0002127647201013734 & lr-AdamW/pg2‚0.0002127647201013734 trainer/global_step‚1299 $ _timestamp‚1.784487178081729e+09 _runtime‚ 654.656680669k4ÉÆà & lr-AdamW/pg1‚0.0002127647201013734 & lr-AdamW/pg2‚0.0002127647201013734 trainer/global_step‚1299 $ _timestamp‚1.784487178081729e+09 _runtime‚ 654.656680669 _step‚101eŸ3Ïr0Ç+ _wandbruntime‚654  _runtime‚654öjMnÈÅO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23043899 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13046153 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6792850 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538869760 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚79439408489472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241367 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚78166022420992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923314 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7844871780825202e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1234732960256 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23054042 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3383552 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13056294 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚98416278454784 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1273537372672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚98416278454784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 _step‚102 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚99651028933632 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 _runtime‚ 654.65736975 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214493696 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358303744 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538869760 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9997748 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3383272 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚79439559793664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9997748 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99650670629888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1273410492928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13046153 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358303744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9997746 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚97388841408400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1234392175104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚98416295973376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10176122 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1234732960256±ÎWÏ'ÉÉO XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚99651028933632 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 _runtime‚ 654.65736975 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214493696 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358303744 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538869760 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9997748 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3383272 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚79439559793664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9997748 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99650670629888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1273410492928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13046153 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358303744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9997746 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚97388841408400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1234392175104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚98416295973376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10176122 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1234732960256 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23043899 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13046153 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6792850 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538869760 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚79439408489472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241367 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚78166022420992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923314 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7844871780825202e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1234732960256 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23054042 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3383552 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13056294 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚98416278454784 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1273537372672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚98416278454784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 _step‚102fKôì80Ê+ _wandbruntime‚655  _runtime‚655û~_”Ù&ËÓM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10005250 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 _step‚103 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358310912 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10005248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001079 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10005248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚830 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23061276 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13066183 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13056028 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128969728 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚98695715004589 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1235322086400 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚79499401263104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23061276 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚99726150621184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1274502509568 trainer/global_step‚1299 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1233430474349 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚98490487742464 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10005250 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10183702 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538869760 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6797931 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923426 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚78224874329088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97462267490880 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153394176 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214493696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1235662878720 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23071433 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3385771 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚79499247868928 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚99726150621184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13066183 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1235662878720 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6797933 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚78224898753536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10183994 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3386061 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23071433 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792320 1íO%Ì×M RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10183994 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3386061 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23071433 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792320 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10005250 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 _step‚103 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358310912 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10005248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001079 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10005248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚830 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23061276 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13066183 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13056028 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128969728 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚98695715004589 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1235322086400 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚79499401263104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23061276 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚99726150621184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1274502509568 trainer/global_step‚1299 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚486 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1233430474349 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚98490487742464 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10005250 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10183702 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538869760 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6797931 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923426 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚78224874329088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97462267490880 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153394176 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214493696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1235662878720 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23071433 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3385771 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚79499247868928 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚99726150621184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13066183 YGDeviceStatsMonitor.on_train_‚3üŽbatch_end/active_bytes.small_pool.allocated‚ 1235662878720 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6797933 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚78224898753536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675g1Âòý0Í+ _wandbruntime‚655  _runtime‚655)¿ôÎü 1loss/FBt2/assignment_loss‚0.33894859693899093 /loss/FRt1/detection_loss‚1.2951818946795381 /loss/FRt2/detection_loss‚0.5461471247558113 1loss/SRqqt2/detection_loss‚1.4541264878442655 _step‚104 2loss/SRqqt1/assignment_loss‚0.7097828536747589 -loss/FBt2/detection_loss‚2.36193980416283 trainer/global_step‚1299 _runtime‚ 655.00702867 0loss/FRt1/assignment_loss‚1.6639726857032895 /loss/FRt2/assignment_loss‚3.411351365601365 1loss/SRqqt2/assignment_loss‚1.163599905541513 1loss/FBt1/assignment_loss‚0.13948919858015152 1loss/SRqqt1/detection_loss‚0.8817765329001881 /loss/FBt1/detection_loss‚1.0967385834749313 % _timestamp‚1.7844871784322753e+09 &loss/total_loss‚15.063055033857633 epoch‚1gÙlæÏ€ /loss/FRt2/detection_loss‚0.5461471247558113 1loss/SRqqt2/detection_loss‚1.4541264878442655 _step‚104 2loss/SRqqt1/assignment_loss‚0.7097828536747589 -loss/FBt2/detection_loss‚2.36193980416283 trainer/global_step‚1299 _runtime‚ 655.00702867 0loss/FRt1/assignment_loss‚1.6639726857032895 /loss/FRt2/assignment_loss‚3.411351365601365 1loss/SRqqt2/assignment_loss‚1.163599905541513 1loss/FBt1/assignment_loss‚0.13948919858015152 1loss/SRqqt1/detection_loss‚0.8817765329001881 /loss/FBt1/detection_loss‚1.0967385834749313 % _timestamp‚1.7844871784322753e+09 &loss/total_loss‚15.063055033857633 epoch‚1 1loss/FBt2/assignment_loss‚0.33894859693899093 /loss/FRt1/detection_loss‚1.2951818946795381hšD3GЂ(:¼ ºôÒÞ¦¾Ë) disk./.usagePercent‚76.54209854431599% disk.nvme1n1.out‚1938.57470703125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4.3125' disk.nvme0c0n1.out‚8843.76220703125 disk.nvme0n1.in‚4.3125% disk.nvme0n1.out‚8843.76220703125! proc.memory.availableMB‚491.5 disk.nvme1n1.in‚0 disk.md0.in‚0! disk.dm-0.out‚9700.5576171875! disk.dm-1.out‚1036.8076171875 disk.rbd0.out‚0# memory_percent‚97.0001220703125# proc.memory.rssMB‚ 2837.51953125) proc.memory.percent‚17.31884479522705 network.sent‚ 1.523803e+06% disk./.usageGB‚1175.1125679016113 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚1938.57470703125 disk.dm-0.in‚4.3125 proc.cpu.threads‚17 network.recv‚223338 disk.md0.out‚ 21.17578125 disk.dm-1.in‚0 cpu‚26.171278416572402 ù8ŸžÑ‚(:“ ºôÒ“è÷Ë gpu.0.gpu‚78.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚56.0 gpu.0.powerWatts‚227.794' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚75.93133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªÞ5ÛKÒ‚(:À žºôÒ´‚¾Ë( proc.memory.availableMB‚ 491.47265625 proc.cpu.threads‚17% disk./.usageGB‚1175.1128959655762 disk.dm-0.in‚4.3125$ disk.nvme0n1.out‚8845.7744140625& disk.nvme1c1n1.out‚1939.0888671875# proc.memory.rssMB‚ 2837.51953125) proc.memory.percent‚17.31884479522705 cpu‚26.162468760249435 disk.nvme1n1.in‚0 disk.md0.out‚ 21.52734375! disk.dm-0.out‚9702.2744140625 disk.dm-1.in‚0! disk.dm-1.out‚1036.8779296875& disk.nvme0c0n1.out‚8845.7744140625 disk.nvme1c1n1.in‚0$ memory_percent‚97.00028896331787 network.recv‚227343$ disk.nvme1n1.out‚1939.0888671875 disk.nvme0c0n1.in‚4.3125 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚ 1.584964e+06) disk./.usagePercent‚76.54211991308101 disk.md0.in‚0 disk.nvme0n1.in‚4.31254Ž pžÓ‚(:“ žºôÒÀåƒÌ gpu.0.gpu‚74.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚56.0 gpu.0.powerWatts‚258.569' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.18966666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08DÆ*0Ô+ _wandbruntime‚677  _runtime‚677I‚}ÇÕÁ ' lr-AdamW/pg2‚0.00021274689511325168 trainer/global_step‚1349 $ _timestamp‚1.784487201151371e+09 _runtime‚ 677.726763401 _step‚105 ' lr-AdamW/pg1‚0.00021274689511325168ÛÁ›ËÖÅ $ _timestamp‚1.784487201151371e+09 _runtime‚ 677.726763401 _step‚105 ' lr-AdamW/pg1‚0.00021274689511325168 ' lr-AdamW/pg2‚0.00021274689511325168 trainer/global_step‚1349i—YM‘0×+ _wandbruntime‚677  _runtime‚677ÿ$§6Û'ØÕO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1280919996928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10556176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚101060145532400 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1281260782080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214493696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1321505679360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1321378799616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1280919996928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚103406786906112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚103407145209856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962674 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 trainer/global_step‚1349 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538869760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚102125866909184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23914601 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10372848 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001079 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7046900 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚102125884427776 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241367 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1278945316009 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23924744 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7046902 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 $ _timestamp‚1.784487201151973e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3508988 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚ 677.727001252 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358303744 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚102125866909184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10555888 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3509274 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚102338750925095 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1278605392695 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126879744 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 _step‚106 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚103406786906112 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13541755 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13541755 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214493696 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151304192 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82431193423360 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785152 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13551896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923314 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10372846 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚81109814623744@J(Àß'ÙÙO IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚103406786906112 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13541755 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13541755 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214493696 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151304192 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82431193423360 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785152 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13551896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923314 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10372846 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚81109814623744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1280919996928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10556176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚101060145532400 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1281260782080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214493696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1321505679360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1321378799616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1280919996928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚103406786906112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚103407145209856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962674 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 trainer/global_step‚1349 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538869760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚102125866909184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23914601 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10372848 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001079 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7046900 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚102125884427776 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241367 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1278945316009 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23924744 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7046902 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 $ _timestamp‚1.784487201151973e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3508988 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚ 677.727001252 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358303744 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚102125866909184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10555888 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3509274 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚102338750925095 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1278605392695 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126879744 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 _step‚106jÒ¥«¨0Ú+ _wandbruntime‚678  _runtime‚678ûÖ7¨ ÛàM EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 678.075913176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1282190700544 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962786 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚101133571614880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23942135 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214493696 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚102200058678272 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1322341838336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923426 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10563549 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7051983 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10563833 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10380348 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1281849908224 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13561785 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚103482266897408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10380348 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82491179897344 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚81168715380736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122678272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23942135 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13561785 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚102200076196864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚81168690956288 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23931978 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1281849908224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 147102720 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚82491032794624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13551630 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10380350 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 trainer/global_step‚1349 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚102200058678272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538869760 RBEÅœÆ DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538869760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1282190700544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚102200076196864 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3511568 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1279533615931 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358310912 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 _step‚107 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚102413462193597 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1279873539357 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7051981 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10380350 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23931978 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13551630 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 % _timestamp‚1.7844872015008144e+09iôê&ÜäM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358310912 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 _step‚107 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚102413462193597 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1279873539357 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7051981 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10380350 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23931978 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13551630 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 % _timestamp‚1.7844872015008144e+09 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 678.075913176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1282190700544 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962786 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚101133571614880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23942135 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214493696 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚102200058678272 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1322341838336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923426 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10563549 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7051983 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10563833 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10380348 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1281849908224 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13561785 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚103482266897408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10380348 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82491179897344 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚81168715380736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122678272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23942135 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13561785 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚102200076196864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚81168690956288 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23931978 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1281849908224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 147102720 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚82491032794624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13551630 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10380350 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 trainer/global_step‚1349 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚102200058678272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538869760 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538869760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1282190700544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚102200076196864 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3511568 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1279533615931 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2k¨“µm0Ý+ _wandbruntime‚678  _runtime‚678úäßÌÞþ 0loss/FRt2/assignment_loss‚3.2779879564370953 /loss/FBt2/assignment_loss‚0.301350699204022 /loss/FRt1/detection_loss‚1.4291760049127828 &loss/total_loss‚14.976843867688725 /loss/FRt1/assignment_loss‚1.897547675643051 /loss/FRt2/detection_loss‚0.5920830039717818 1loss/SRqqt1/detection_loss‚0.7442618283179647 2loss/SRqqt1/assignment_loss‚0.8678217735947468 2loss/SRqqt2/assignment_loss‚0.9300362518181403 1loss/SRqqt2/detection_loss‚1.5460630789895853 epoch‚1 % _timestamp‚1.7844872015016816e+09 1loss/FBt1/assignment_loss‚0.16456162193241133 /loss/FBt1/detection_loss‚1.1200454229442254 /loss/FBt2/detection_loss‚2.1059085499229178 trainer/global_step‚1349 _runtime‚ 678.076424329 _step‚108E.œŽß‚ /loss/FRt1/assignment_loss‚1.897547675643051 /loss/FRt2/detection_loss‚0.5920830039717818 1loss/SRqqt1/detection_loss‚0.7442618283179647 2loss/SRqqt1/assignment_loss‚0.8678217735947468 2loss/SRqqt2/assignment_loss‚0.9300362518181403 1loss/SRqqt2/detection_loss‚1.5460630789895853 epoch‚1 % _timestamp‚1.7844872015016816e+09 1loss/FBt1/assignment_loss‚0.16456162193241133 /loss/FBt1/detection_loss‚1.1200454229442254 /loss/FBt2/detection_loss‚2.1059085499229178 trainer/global_step‚1349 _runtime‚ 678.076424329 _step‚108 0loss/FRt2/assignment_loss‚3.2779879564370953 /loss/FBt2/assignment_loss‚0.301350699204022 /loss/FRt1/detection_loss‚1.4291760049127828 &loss/total_loss‚14.976843867688725l d´Hà‚(:½ ­ºôÒÑŒÌ) disk./.usagePercent‚76.56183284727841 disk.dm-0.in‚4.3125 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0# proc.memory.rssMB‚ 2837.51953125 cpu‚26.15185525316177' disk.nvme0c0n1.out‚8891.12548828125' disk.nvme1c1n1.out‚1940.65478515625% disk.nvme1n1.out‚1940.65478515625 proc.cpu.threads‚17 network.sent‚ 1.603748e+06 network.recv‚230751% disk./.usageGB‚1175.4155387878418 disk.md0.out‚ 21.9765625 disk.dm-0.out‚9748.060546875 disk.dm-1.out‚ 1037.06640625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.md0.in‚0 disk.nvme0c0n1.in‚4.3125 disk.nvme0n1.in‚4.3125% disk.nvme0n1.out‚8891.12548828125$ memory_percent‚99.96316432952881& proc.memory.availableMB‚ 6.03515625) proc.memory.percent‚17.31884479522705ߺ1¢žá‚(:“ ­ºôÒåܼÌ gpu.0.gpu‚82.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚56.0 gpu.0.powerWatts‚245.175' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚81.72500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ãõm¡0â+ _wandbruntime‚701  _runtime‚701´ØÔŸÈã ' lr-AdamW/pg1‚0.00021272736715823772 ' lr-AdamW/pg2‚0.00021272736715823772 trainer/global_step‚1399 % _timestamp‚1.7844872247095962e+09 _runtime‚ 701.284581829 _step‚109)ÌäÆ % _timestamp‚1.7844872247095962e+09 _runtime‚ 701.284581829 _step‚109 ' lr-AdamW/pg1‚0.00021272736715823772 ' lr-AdamW/pg2‚0.00021272736715823772 trainer/global_step‚1399m¹Ãsd0å+ _wandbruntime‚701  _runtime‚701=sÂðÛ'æÕO G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10747948 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10935781 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1369373539840 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10747946 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131074048 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155498496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 % _timestamp‚1.7844872247104971e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10747946 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3634829 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1325388381017 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538869760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10935501 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105835472882176 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24795446 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚84053655675392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24785303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚85423004790784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1327447818752 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14047498 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1369504613888 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104731466695760 _step‚110 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1327788603904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚84053631250944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104731449656400 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14037357 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10747948 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214493696 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1327447818752 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7300950 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚1399 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚107163261486080 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358303744 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚106056855076777 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241367 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105835455363584 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚107162903182336 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚85423160289280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001079 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14047498 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚107162903182336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214493696 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚106056498114103 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 _runtime‚ 701.28534221 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105835455363584 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3634551 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962674 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358303744 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962674 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358303744 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10747948 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10935781 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1369373539840 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10747946 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131074048 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155498496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 % _timestamp‚1.7844872247104971e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10747946 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3634829 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1325388381017 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538869760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10935501 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105835472882176 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24795446 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚84053655675392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24785303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚85423004790784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1327447818752 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14047498 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1369504613888 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104731466695760 _step‚110 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_bH seÛatch_start/active.large_pool.peak‚1446 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1327788603904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚84053631250944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104731449656400 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14037357 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10747948 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214493696 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1327447818752 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7300950 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚1399 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚107163261486080 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358303744 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚106056855076777 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241367 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105835455363584 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚107162903182336 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚85423160289280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001079 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14047498 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚107162903182336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214493696 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚106056498114103 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 _runtime‚ 701.28534221 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105835455363584 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3634551n£´µŽ0è+ _wandbruntime‚701  _runtime‚701B©Qå&éßM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10755450 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7306031 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538869760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚107238024862720 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚85482996646912 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10943022 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7306033 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚85482836961280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚84112532007936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚107238383173632 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962786 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10755448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚107238383173632 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1328377730048 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10943297 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135261184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10755450 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 _step‚111 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105909647132672 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792320 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792320 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14057387 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14047232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159685632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105909664651264 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923426 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 _runtime‚ 701.643790908 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105909664651264 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24802680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3636991 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1370329377792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚106130852419819 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚104804875738880 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚84112507583488 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚106131209382605 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1325976680939 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14057387 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214493696 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24802680 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14047232 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1328718522368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1370464638976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24812837 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358310912 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚105909647132672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1328718522368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1328377730048 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24812837 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675ÔBñ*é&êãM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538869760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚107238024862720 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚85482996646912 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10943022 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7306033 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚85482836961280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚84112532007936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚107238383173632 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962786 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10755448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚107238383173632 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1328377730048 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10943297 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135261184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10755450 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 _step‚111 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105909647132672 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792320 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792320 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14057387 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14047232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159685632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105909664651264 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923426 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 _runtime‚ 701.643790908 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105909664651264 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24802680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3636991 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1370329377792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚106130852419819 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚104804875738880 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚84112507583488 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚106131209382605 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1325976680939 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14057387 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214493696 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24802680 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14047232 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1328718522368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1370464638976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24812837 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358310912 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚105909647132672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1328718522368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1328377730048 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24812837 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10755450 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7306031 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0o ©>50ë+ _wandbruntime‚701  _runtime‚701‚Æ¥ðì€ trainer/global_step‚1399 _runtime‚ 701.644219329 0loss/FRt1/assignment_loss‚1.7455293342034282 1loss/FBt1/assignment_loss‚0.18956389291912407 /loss/FRt1/detection_loss‚1.2921341305546548 1loss/SRqqt1/detection_loss‚0.9180807384447867 2loss/SRqqt2/assignment_loss‚1.0322695952087928 % _timestamp‚1.7844872250695827e+09 _step‚112 /loss/FRt2/assignment_loss‚3.121176437859319 1loss/FBt2/assignment_loss‚0.27297495398961474 /loss/FBt1/detection_loss‚1.6391330362837104 /loss/FBt2/detection_loss‚1.3846834510564805 &loss/total_loss‚14.449517695325579 2loss/SRqqt1/assignment_loss‚0.7386603346121562 /loss/FRt2/detection_loss‚0.6419885353474747 1loss/SRqqt2/detection_loss‚1.4733232548460364 epoch‚1˜U ” í„ &loss/total_loss‚14.449517695325579 2loss/SRqqt1/assignment_loss‚0.7386603346121562 /loss/FRt2/detection_loss‚0.6419885353474747 1loss/SRqqt2/detection_loss‚1.4733232548460364 epoch‚1 trainer/global_step‚1399 _runtime‚ 701.644219329 0loss/FRt1/assignment_loss‚1.7455293342034282 1loss/FBt1/assignment_loss‚0.18956389291912407 /loss/FRt1/detection_loss‚1.2921341305546548 1loss/SRqqt1/detection_loss‚0.9180807384447867 2loss/SRqqt2/assignment_loss‚1.0322695952087928 % _timestamp‚1.7844872250695827e+09 _step‚112 /loss/FRt2/assignment_loss‚3.121176437859319 1loss/FBt2/assignment_loss‚0.27297495398961474 /loss/FBt1/detection_loss‚1.6391330362837104 /loss/FBt2/detection_loss‚1.3846834510564805p­Fî‚(:» ¼ºôÒ¥â€Ì% disk.nvme1n1.out‚1978.36572265625 disk.dm-1.in‚0# memory_percent‚99.9650239944458& proc.memory.availableMB‚ 5.73046875# proc.memory.rssMB‚ 2837.51953125 disk.rbd0.in‚0 disk.dm-1.out‚ 1037.19921875 cpu‚26.143478418606517 proc.cpu.threads‚17 disk.nvme0c0n1.in‚4.3125' disk.nvme0c0n1.out‚9018.93115234375 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚1978.36572265625 disk.md0.in‚0) proc.memory.percent‚17.31884479522705 network.sent‚ 1.652329e+06% disk./.usageGB‚1175.3512573242188 disk.nvme0n1.in‚4.3125% disk.nvme0n1.out‚9018.93115234375 disk.nvme1n1.in‚0 disk.md0.out‚22.5625 disk.dm-0.in‚4.3125! disk.dm-0.out‚9912.1943359375 network.recv‚235194) disk./.usagePercent‚76.55764581170615 disk.rbd0.out‚0)HQï‚(:’ ¼ºôÒ™ÈÌ gpu.0.gpu‚22.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚231.562' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚77.18733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0À]~Ið‚(:¾ ˺ôÒïÏË! disk.dm-0.out‚9982.8857421875" disk.dm-1.out‚1037.35107421875 disk.rbd0.in‚0# proc.memory.rssMB‚ 2837.51953125) proc.memory.percent‚17.31884479522705 cpu‚26.135805101031952 network.recv‚239393' disk.nvme0c0n1.out‚9059.50244140625 proc.cpu.threads‚17 network.sent‚ 1.714756e+06) disk./.usagePercent‚76.55764953881634% disk.nvme0n1.out‚9059.50244140625$ disk.nvme1n1.out‚2009.1318359375 disk.md0.in‚0 disk.dm-0.in‚4.3125 disk.nvme0c0n1.in‚4.3125% disk./.usageGB‚1175.3513145446777 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.out‚22.78125 disk.dm-1.in‚0 disk.rbd0.out‚0$ memory_percent‚99.96354579925537& proc.memory.availableMB‚ 5.97265625 disk.nvme0n1.in‚4.3125& disk.nvme1c1n1.out‚2009.1318359375ÄE1“ñ‚(:ˆ ˺ôÒ•¥’Ì gpu.0.gpu‚71.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚266.904' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚88.968 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#µ–0ò+ _wandbruntime‚725  _runtime‚725k¥cÆóÀ trainer/global_step‚1449 % _timestamp‚1.7844872486074402e+09 _runtime‚ 725.182420872 _step‚113 & lr-AdamW/pg1‚0.0002127061365491114 & lr-AdamW/pg2‚0.0002127061365491114¥²ßqÊôÄ trainer/global_step‚1449 % _timestamp‚1.7844872486074402e+09 _runtime‚ 725.182420872 _step‚113 & lr-AdamW/pg1‚0.0002127061365491114 & lr-AdamW/pg2‚0.0002127061365491114qY·«S0õ+ _wandbruntime‚725  _runtime‚725“l¿¨ÈöÖO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14532959 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1371491522711 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚86997447878144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962674 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538869760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚110919377762304 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11123046 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1373975640576 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241367 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚108402770819760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14543100 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785152 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14543100 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚109545061336576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1417475875328 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚110919019458560 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214493696 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1374316425728 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7555000 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3760539 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25656005 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358303744 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 ?9DeviceStatsMonitor.on_train_batch_start/segment.Âé®all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11123048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11315541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538869760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128976896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7844872486083193e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚110919019458560 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214493696 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1374316425728 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7555000 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3760539 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25656005 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358303744 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11123048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11315541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538869760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128976896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7844872486083193e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚86997447878144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962674 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538869760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚110919377762304 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11123046 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1373975640576 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241367 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚108402770819760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14543100 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785152 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14543100 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚109545061336576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1417475875328rCÀm¹0ø+ _wandbruntime‚725  _runtime‚725m¼Pä&ùÞM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚110994141138944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1374905551872 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚110994499449856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚108476196902240 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1372419745947 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11130548 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 _step‚115 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109619253105664 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1418437268480 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923426 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25673382 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001079 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214493696 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1375246344192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14542834 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚295 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14542834 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1374905551872 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7560083 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3763062 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1449 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25683539 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚87056348635136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚109848956571613 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538869760 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25673382 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14552989 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133164032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚109619235587072 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3762769 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚108476179862880 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11130550 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11322850 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1418304104448 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14552989 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚110994499449856 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11323145 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25683539 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11130550 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚88474628315136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚109848599608827 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109619253105664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157588480 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 ܾâÕúâM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538869760 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25673382 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14552989 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133164032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚109619235587072 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3762769 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚108476179862880 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11130550 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11322850 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1418304104448 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14552989 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚110994499449856 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11323145 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25683539 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11130550 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚88474628315136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚109848599608827 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109619253105664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157588480 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚110994141138944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1374905551872 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚110994499449856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚108476196902240 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1372419745947 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11130548 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 _step‚115 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109619253105664 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1418437268480 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923426 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25673382 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.s?@µÚ mall_pool.peak‚ 538001079 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214493696 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1375246344192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14542834 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚295 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14542834 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1374905551872 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7560083 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3763062 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1449 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25683539 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚87056348635136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚109848956571613 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960sìÝæ0û+ _wandbruntime‚725  _runtime‚725ª}¾6üþ _step‚116 /loss/FRt2/assignment_loss‚3.149640297972112 2loss/SRqqt1/assignment_loss‚0.7545908404932614 /loss/FRt2/detection_loss‚0.6355772520383833 1loss/SRqqt2/detection_loss‚1.4265258707079693 _runtime‚ 725.541379714 2loss/SRqqt2/assignment_loss‚0.7530213507843462 1loss/FBt2/assignment_loss‚0.08415688852255698 /loss/FRt1/detection_loss‚1.3384127422443914 1loss/SRqqt1/detection_loss‚0.8117269880164942 &loss/total_loss‚23.232763222324827 0loss/FRt1/assignment_loss‚1.7072722873102457 /loss/FBt1/detection_loss‚1.6782760389293694 epoch‚1 trainer/global_step‚1449 1loss/FBt1/assignment_loss‚0.19038720252331487 /loss/FBt2/detection_loss‚10.703175462782383 # _timestamp‚1.78448724896672e+09ãm¡ý‚ 1loss/FBt1/assignment_loss‚0.19038720252331487 /loss/FBt2/detection_loss‚10.703175462782383 # _timestamp‚1.78448724896672e+09 _step‚116 /loss/FRt2/assignment_loss‚3.149640297972112 2loss/SRqqt1/assignment_loss‚0.7545908404932614 /loss/FRt2/detection_loss‚0.6355772520383833 1loss/SRqqt2/detection_loss‚1.4265258707079693 _runtime‚ 725.541379714 2loss/SRqqt2/assignment_loss‚0.7530213507843462 1loss/FBt2/assignment_loss‚0.08415688852255698 /loss/FRt1/detection_loss‚1.3384127422443914 1loss/SRqqt1/detection_loss‚0.8117269880164942 &loss/total_loss‚23.232763222324827 0loss/FRt1/assignment_loss‚1.7072722873102457 /loss/FBt1/detection_loss‚1.6782760389293694 epoch‚1 trainer/global_step‚1449t…¥Ù%Oþ‚(:Ä ÚºôÒ¬ˆàË) disk./.usagePercent‚76.57507328194335 disk.nvme1c1n1.in‚0 disk.md0.out‚ 23.0859375 cpu‚26.127734283689534 network.sent‚ 1.720589e+06# disk.nvme1n1.out‚2017.853515625 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0" disk.dm-1.out‚1037.63232421875 disk.nvme0c0n1.in‚ 4.3359375# proc.memory.rssMB‚ 2837.51953125) proc.memory.percent‚17.31884479522705 network.recv‚242852% disk.nvme1c1n1.out‚2017.853515625 disk.nvme1n1.in‚0 disk.md0.in‚0! disk.dm-0.out‚9991.6435546875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 4.3359375$ memory_percent‚99.96395111083984% disk./.usageGB‚1175.6188125610352 disk.rbd0.out‚0' disk.nvme0c0n1.out‚9060.48974609375% disk.nvme0n1.out‚9060.48974609375# proc.memory.availableMB‚5.90625 proc.cpu.threads‚17ˆ»Î³Ÿÿ‚(:” ÚºôÒ¬ˆÀÌ gpu.0.gpu‚38.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.621' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.873666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?ºt0€+ _wandbruntime‚749  _runtime‚749'Å9ľ _step‚117 % lr-AdamW/pg1‚0.000212683203625924 % lr-AdamW/pg2‚0.000212683203625924 trainer/global_step‚1499 % _timestamp‚1.7844872724908757e+09 _runtime‚ 749.067407163HÍ,È‚Â trainer/global_step‚1499 % _timestamp‚1.7844872724908757e+09 _runtime‚ 749.067407163 _step‚117 % lr-AdamW/pg1‚0.000212683203625924 % lr-AdamW/pg2‚0.000212683203625924u§‘Ï0ƒ+ _wandbruntime‚749  _runtime‚749KˆÁéÜ'„ÖO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113491992492119 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26526707 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11498146 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1499 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚113254649790976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7809050 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11498148 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131074048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1465325236224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1417934587719 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214493696 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1420503462400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚114675494038528 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3886860 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 _step‚118 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15028561 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3886566 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001079 _runtime‚ 749.067462403 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚114675135734784 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15028561 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 % _timestamp‚1.7844872724914849e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1420844247552 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26536850 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143q¶îà'…ÚO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚89941264505344 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113491992492119 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26526707 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11498146 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1499 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚113254649790976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7809050 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11498148 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131074048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1465325236224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1417934587719 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214493696 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1420503462400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚114675494038528 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3886860 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 _step‚118 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15028561 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3886566 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001079 _runtime‚ 749.067462403 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚114675135734784 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15028561 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 % _timestamp‚1.7844872724914849e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1420844247552 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26536850 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1419202734381 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7814133 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3889421 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91466427900416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90000140837888 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1466413935104 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚112147483986880 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792320 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11505650 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1421774166016 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 _runtime‚ 749.423267769 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26544084 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113566703760621 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538869760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11703554 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚112147501026240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792320 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15048591 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1421433373696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 _step‚119 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1421433373696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚114750615726080 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214493696 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1418862810955 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15038436 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1421774166016 HBDeviceStatsMonitor.on_trainxl·‰\_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538869760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚114750257415168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11505648 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151297024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126872576 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962786 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113566346797835 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26554241 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26544084 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001079 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90000165262336 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923426 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7814131 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚113328841560064 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15048591 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26554241 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1466287062528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11703269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11505650 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11505648 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚113328824041472 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚113328824041472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 $ _timestamp‚1.784487272848137e+09Ð/¶³é&ˆãM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26554241 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1466287062528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11703269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11505650 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11505648 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚113328824041472 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚113328824041472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 $ _timestamp‚1.784487272848137e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚114750615726080 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1419202734381 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7814133 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3889421 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91466427900416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90000140837888 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1466413935104 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚112147483986880 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792320 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11505650 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1421774166016 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 _runtime‚ 749.423267769 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26544084 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113566703760621 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538869760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11703554 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚112147501026240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792320 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15048591 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1421433373696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 _step‚119 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1421433373696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚114750615726080 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214493696 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1418862810955 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15038436 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1421774166016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538869760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚114750257415168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11505648 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151297024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126872576 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962786 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113566346797835 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26554241 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26544084 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001079 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90000165262336 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923426 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7814131 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚113328841560064 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15048591wðæIà0‰+ _wandbruntime‚749  _runtime‚7497Af©Šÿ 1loss/FBt1/assignment_loss‚0.08311019692188663 _step‚120 1loss/FBt2/assignment_loss‚0.16289530099416688 /loss/FRt2/detection_loss‚0.5688176082726568 1loss/SRqqt2/detection_loss‚1.4645992582009058 /loss/FBt1/detection_loss‚0.9410925992298456 %loss/total_loss‚12.40241171947052 0loss/FRt2/assignment_loss‚3.1067980324520783 epoch‚1 trainer/global_step‚1499 2loss/SRqqt1/assignment_loss‚0.7221282860218656 /loss/FRt1/detection_loss‚1.3014012896797271 1loss/SRqqt1/detection_loss‚0.7895333000598246 /loss/FBt2/detection_loss‚1.1276338225916813 % _timestamp‚1.7844872728490922e+09 _runtime‚ 749.42376085 0loss/FRt1/assignment_loss‚1.4850662701270159 2loss/SRqqt2/assignment_loss‚0.6493357549188659bü ‹ƒ 2loss/SRqqt1/assignment_loss‚0.7221282860218656 /loss/FRt1/detection_loss‚1.3014012896797271 1loss/SRqqt1/detection_loss‚0.7895333000598246 /loss/FBt2/detection_loss‚1.1276338225916813 % _timestamp‚1.7844872728490922e+09 _runtime‚ 749.42376085 0loss/FRt1/assignment_loss‚1.4850662701270159 2loss/SRqqt2/assignment_loss‚0.6493357549188659 1loss/FBt1/assignment_loss‚0.08311019692188663 _step‚120 1loss/FBt2/assignment_loss‚0.16289530099416688 /loss/FRt2/detection_loss‚0.5688176082726568 1loss/SRqqt2/detection_loss‚1.4645992582009058 /loss/FBt1/detection_loss‚0.9410925992298456 %loss/total_loss‚12.40241171947052 0loss/FRt2/assignment_loss‚3.1067980324520783 epoch‚1 trainer/global_step‚1499xo„»_DŒ‚(:¹ éºôÒ¿—áË' disk.nvme0c0n1.out‚9247.73193359375! disk.nvme1c1n1.out‚ 2794.09375 cpu‚26.120643809612346 proc.cpu.threads‚17 disk.md0.out‚23.4375 disk.nvme0c0n1.in‚ 4.3359375% disk.nvme0n1.out‚9247.73193359375" memory_percent‚99.965500831604 network.sent‚ 1.782863e+06% disk./.usageGB‚1176.2975692749023 disk.nvme1n1.out‚ 2794.09375 disk.dm-0.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0) proc.memory.percent‚17.31884479522705 network.recv‚246916 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-0.out‚10953.94140625 disk.rbd0.in‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 5.65234375# proc.memory.rssMB‚ 2837.51953125) disk./.usagePercent‚76.61928475980474 disk.dm-1.in‚0" disk.dm-1.out‚1038.06103515625­Ä@ž‚(:“ éºôÒ€Ü¾Ì gpu.0.gpu‚97.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚56.0 gpu.0.powerWatts‚267.968' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.32266666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#·ñeSŽ‚(:È øºôÒ«±ØË network.sent‚ 1.845112e+06% disk./.usageGB‚1176.3011474609375! disk.dm-0.out‚10954.982421875 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚2795.0751953125$ memory_percent‚99.96426105499268& proc.memory.availableMB‚ 5.85546875 proc.cpu.threads‚17 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0' disk.nvme0c0n1.out‚9249.13720703125& disk.nvme1c1n1.out‚2795.0751953125) proc.memory.percent‚17.31884479522705 cpu‚26.11351089437471) disk./.usagePercent‚76.61951782842779 disk.rbd0.out‚0 disk.nvme0n1.in‚ 4.3359375% disk.nvme0n1.out‚9249.13720703125 disk.md0.out‚ 23.9453125 network.recv‚251368" disk.dm-1.out‚1038.33447265625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0 disk.md0.in‚0# proc.memory.rssMB‚ 2837.51953125Ìü[Ÿ‚(:” øºôÒϯœÌ gpu.0.gpu‚85.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.888' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.629333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡Ožì0+ _wandbruntime‚772  _runtime‚772ËbUiÇ‘Á ' lr-AdamW/pg1‚0.00021265856875599286 ' lr-AdamW/pg2‚0.00021265856875599286 trainer/global_step‚1549 $ _timestamp‚1.784487296027513e+09 _runtime‚ 772.602516168 _step‚1212åjrË’Å ' lr-AdamW/pg1‚0.00021265856875599286 ' lr-AdamW/pg2‚0.00021265856875599286 trainer/global_step‚1549 $ _timestamp‚1.784487296027513e+09 _runtime‚ 772.602516168 _step‚121y(RW0“+ _wandbruntime‚772  _runtime‚772h²/#Ü'”ÖO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚116964220726784 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 _runtime‚ 772.606375492 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4012882 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚92885081132544 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1464717576041 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214493696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1513469090304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11873248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 _step‚122 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785152 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538869760 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚118431610314752 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚116964238245376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8063100 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4013167 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15524163 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚118431252011008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12076269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27397409 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1513469090304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11873248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 _step‚122 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785152 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538869760 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚118431610314752 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚116964238245376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8063100 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4013167 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15524163 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚118431252011008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12076269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27397409 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1464717576041z˜rù@0–+ _wandbruntime‚772  _runtime‚772ßz¤æ&—àM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚117284093986843 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27424943 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538869760 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12083633 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1465645799389 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15544193 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7844872963834043e+09 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚118506732002304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27414786 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923426 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1465305875963 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚ 772.958536938 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11880750 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1467961195520 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚94458251765760 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1467961195520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11880748 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92943981889536 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1514427464704 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214493696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚92943957465088 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792320 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚117038430014464 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001079 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8068183 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚115818788110880 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962786 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚115818805150240 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11880748 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27424943 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4015718 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15534038 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792320 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538869760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 trainer/global_step‚1549 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1468301987840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133164032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚118506732002304 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚117038412495872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358310912 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4015452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 _step‚123ÿê&˜äM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1467961195520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11880748 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92943981889536 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1514427464704 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214493696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚92943957465088 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792320 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚117038430014464 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001079 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8068183 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚115818788110880 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962786 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚115818805150240 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11880748 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27424943 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4015718 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15534038 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792320 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538869760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 trainer/global_step‚1549 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1468301987840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133164032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚118506732002304 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚117038412495872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358310912 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4015452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 _step‚123 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚117284093986843 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27424943 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538869760 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12083633 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1465645799389 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15544193 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7844872963834043e+09 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚118506732002304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27414786 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923426 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1465305875963 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚ 772.958536938 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11880750 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1467961195520 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚94458251765760{HÍx0™+ _wandbruntime‚772  _runtime‚772D 8šü 0loss/FRt1/assignment_loss‚1.5638754232462508 1loss/SRqqt2/assignment_loss‚0.880189755813877 -loss/FBt2/detection_loss‚1.84013184261615 &loss/total_loss‚13.528152277634714 % _timestamp‚1.7844872963843474e+09 0loss/FRt2/assignment_loss‚2.9918851712247827 0loss/FBt1/assignment_loss‚0.0937289717277343 1loss/FBt2/assignment_loss‚0.26149517834543684 /loss/FRt1/detection_loss‚1.5401157832588495 1loss/SRqqt1/detection_loss‚0.7556875741096939 /loss/FBt1/detection_loss‚1.1061553724993967 1loss/SRqqt2/detection_loss‚1.2750191750256472 _runtime‚ 772.959017519 2loss/SRqqt1/assignment_loss‚0.6735940120917611 .loss/FRt2/detection_loss‚0.546274017675135 epoch‚1 trainer/global_step‚1549 _step‚124µÞIÒ›€ -loss/FBt2/detection_loss‚1.84013184261615 &loss/total_loss‚13.528152277634714 % _timestamp‚1.7844872963843474e+09 0loss/FRt2/assignment_loss‚2.9918851712247827 0loss/FBt1/assignment_loss‚0.0937289717277343 1loss/FBt2/assignment_loss‚0.26149517834543684 /loss/FRt1/detection_loss‚1.5401157832588495 1loss/SRqqt1/detection_loss‚0.7556875741096939 /loss/FBt1/detection_loss‚1.1061553724993967 1loss/SRqqt2/detection_loss‚1.2750191750256472 _runtime‚ 772.959017519 2loss/SRqqt1/assignment_loss‚0.6735940120917611 .loss/FRt2/detection_loss‚0.546274017675135 epoch‚1 trainer/global_step‚1549 _step‚124 0loss/FRt1/assignment_loss‚1.5638754232462508 1loss/SRqqt2/assignment_loss‚0.880189755813877|BGL½Lœ‚(:Á ‡»ôÒ«ÚÜË disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 2796.01171875 cpu‚26.10664036242914) disk./.usagePercent‚76.61951087115546 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 2796.01171875# proc.memory.rssMB‚ 2837.51953125 network.recv‚254578" disk.dm-1.out‚1038.43994140625$ memory_percent‚99.96447563171387% proc.memory.availableMB‚ 5.8203125) proc.memory.percent‚17.31884479522705 proc.cpu.threads‚17 network.sent‚ 1.849904e+06$ disk./.usageGB‚1176.301040649414 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 disk.dm-0.out‚10959.57421875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 disk.md0.out‚ 24.1640625 disk.dm-1.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚9253.37255859375% disk.nvme0n1.out‚9253.372558593750%ƒ÷‚(:’ ‡»ôÒ˱½Ì gpu.0.gpu‚33.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚143.818' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚47.93933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<¡žž‚(:“ –»ôÒ¢µíË gpu.0.gpu‚91.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚252.298' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.09933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06ó)SŸ‚(:È –»ôÒžå„Ì) disk./.usagePercent‚76.61953845177075 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 5.56640625 disk.nvme0n1.in‚ 4.3359375% disk.nvme0n1.out‚9257.87939453125 disk.nvme1c1n1.in‚0 disk.md0.out‚ 24.76953125" disk.dm-1.out‚1038.52197265625$ memory_percent‚99.96602535247803# proc.memory.rssMB‚ 2837.51953125) proc.memory.percent‚17.31884479522705 network.sent‚ 1.912927e+06 network.recv‚258467$ disk.nvme1n1.out‚2796.8271484375 disk.dm-0.out‚10963.55078125 cpu‚26.10062405507048 proc.cpu.threads‚17% disk./.usageGB‚1176.3014640808105 disk.nvme0c0n1.in‚ 4.3359375' disk.nvme0c0n1.out‚9257.87939453125& disk.nvme1c1n1.out‚2796.8271484375 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0 disk.rbd0.in‚0ªŽæ0 + _wandbruntime‚796  _runtime‚796ÔŠœ¤Å¡¿ _runtime‚ 796.465048928 _step‚125 & lr-AdamW/pg1‚0.0002126322323338954 & lr-AdamW/pg2‚0.0002126322323338954 trainer/global_step‚1599 $ _timestamp‚1.784487319890063e+09Æ!ñɢà $ _timestamp‚1.784487319890063e+09 _runtime‚ 796.465048928 _step‚125 & lr-AdamW/pg1‚0.0002126322323338954 & lr-AdamW/pg2‚0.0002126322323338954 trainer/global_step‚1599}]0£+ _wandbruntime‚796  _runtime‚796±Ný"¤ÖO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1513559106048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538869760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4138858 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚120927486870135 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚119416666152400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12456008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚122187726590976 JDDeviceStatsMonitor.on_train_batch_starÍw¸[º"t/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12248348 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1510820717735 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚122187368287232 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚120673809181184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚120673826699776 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 % _timestamp‚1.7844873198909342e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16019765 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4139144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128976896 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923314 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358303744 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153401344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962674 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚120673809181184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1513899891200 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95828897759744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241367 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214493696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1561464163840 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001079 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538869760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16019765 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8317150 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1511160641049 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12456296 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16029906 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8317152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12248346 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚122187368287232 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚120673809181184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚120673826699776 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 % _timestamp‚1.7844873198909342e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16019765 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4139144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128976896 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923314 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358303744 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153401344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962674 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚120673809181184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1513899891200 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95828897759744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241367 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214493696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1561464163840 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001079 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538869760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16019765 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8317150 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1511160641049 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12456296 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16029906 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8317152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12248346 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844873202469049e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8322233 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1562422405120 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923426 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16029640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28285488 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚119490109274240 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1511748940971 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚120748018468864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28295645 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚97450067527680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚121001841175851 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1512088864397 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚122262848278528 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1514489017344 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12255848 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28295645 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358310912 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12463882 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚97450220921856 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚121002198138637 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16039795 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12255850 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128969728 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚120748018468864 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚119490092234880 _step‚127 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 trainer/global_step‚1599 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538869760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚120748000950272 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153394176 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12463599 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001079 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259£³wÃ-¨äM _step‚127 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 trainer/global_step‚1599 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538869760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚120748000950272 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214493696 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153394176 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12463599 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001079 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚122262489967616 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844873202469049e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8322233 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1562422405120 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923426 HBDeviceStatsMonitor.on_train_Ä¸ç ½batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16029640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358310912 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28285488 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚119490109274240 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1511748940971 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚120748018468864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28295645 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚97450067527680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚121001841175851 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1512088864397 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚122262848278528 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1514489017344 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12255848 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241367 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28295645 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358310912 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12463882 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚97450220921856 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚121002198138637 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16039795 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12255850 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128969728 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚120748018468864 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚119490092234880eGÝr0©+ _wandbruntime‚796  _runtime‚796|lï-ªþ epoch‚1 trainer/global_step‚1599 _step‚128 /loss/FBt1/detection_loss‚0.9751574786584043 /loss/FBt2/detection_loss‚1.7811569676679724 $ _timestamp‚1.784487320248009e+09 2loss/SRqqt1/assignment_loss‚0.7730709347604054 /loss/FRt2/detection_loss‚0.5864996511200611 0loss/SRqqt1/detection_loss‚0.769414724385349 _runtime‚ 796.823259192 0loss/FRt1/assignment_loss‚1.7990240119937537 0loss/FRt2/assignment_loss‚2.9059769314783153 1loss/FBt2/assignment_loss‚0.31049765021024917 /loss/FRt1/detection_loss‚1.4501945987736902 2loss/SRqqt2/assignment_loss‚0.7571633319617138 0loss/FBt1/assignment_loss‚0.1738584244223943 1loss/SRqqt2/detection_loss‚1.0348060456587704 &loss/total_loss‚13.316820751091079±zew «ƒ 0loss/FBt1/assignment_loss‚0.1738584244223943 1loss/SRqqt2/detection_loss‚1.0348060456587704 &loss/total_loss‚13.316820751091079 epoch‚1 trainer/global_step‚1599 _step‚128 /loss/FBt1/detection_loss‚0.9751574786584043 /loss/FBt2/detection_loss‚1.7811569676679724 $ _timestamp‚1.784487320248009e+09 2loss/SRqqt1/assignment_loss‚0.7730709347604054 /loss/FRt2/detection_loss‚0.5864996511200611 0loss/SRqqt1/detection_loss‚0.769414724385349 _runtime‚ 796.823259192 0loss/FRt1/assignment_loss‚1.7990240119937537 0loss/FRt2/assignment_loss‚2.9059769314783153 1loss/FBt2/assignment_loss‚0.31049765021024917 /loss/FRt1/detection_loss‚1.4501945987736902 2loss/SRqqt2/assignment_loss‚0.7571633319617138€AFaT¬‚(:É ¥»ôÒ›±øË% disk.nvme1n1.out‚2797.76904296875# proc.memory.rssMB‚ 2841.48828125 cpu‚26.135210541826464 proc.cpu.threads‚17 disk.md0.in‚0' disk.nvme1c1n1.out‚2797.76904296875) proc.memory.percent‚17.34306812286377 network.sent‚ 1.91595e+06) disk./.usagePercent‚76.61949894440289 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚10970.83154296875 disk.dm-1.out‚1038.693359375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1n1.in‚0 network.recv‚261819% disk./.usageGB‚1176.3008575439453 disk.md0.out‚ 25.24609375 disk.nvme0c0n1.in‚ 4.3359375$ disk.nvme0n1.out‚9265.3994140625 disk.nvme1c1n1.in‚0$ memory_percent‚99.99072551727295& proc.memory.availableMB‚ 1.51953125& disk.nvme0c0n1.out‚9265.3994140625 disk.dm-1.in‚0 disk.nvme0n1.in‚ 4.3359375|X™œ­‚(:‘ ¥»ôÒ©‚°Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚52.0 gpu.0.powerWatts‚117.677' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.22566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥2ן0®+ _wandbruntime‚815  _runtime‚815O¿Dø=¯ò{ 8"Purity/2FRt0SRqqt1FBt/event_purity‚0.173611119389534 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9439451098442078 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5957640409469604 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7138746380805969 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.017450423911213875 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7937012910842896 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.14521166682243347 +jet/accuracy_4_of_4‚0.01230158656835556 particle/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4197208285331726 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.012225205078721046 particle/accuracy_2_of_3‚1 *jet/accuracy_2_of_4‚0.7699335813522339 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.11354006081819534 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04215589538216591 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_2‚1 +particle/specificity‚0.9009271264076233 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8364434838294983 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5084346532821655 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.03980943560600281 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8571428656578064 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.06623491644859314 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.2857142984867096 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4199839234352112 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.43201977014541626 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.06354166567325592 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4027476906776428 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6925298571586609 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5875620245933533 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7809703350067139 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7541711926460266 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04385865852236748 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.2857142984867096 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 jet/accuracy_4_of_5‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4328698217868805 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.0626683458685875 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5900972485542297 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1780446618795395 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6000000238418579 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6433731913566589 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5046546459197998 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.14272376894950867 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6514992713928223 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6880733966827393 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5679411888122559 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9885139465332031 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9876542687416077 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4930236339569092 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.1756656914949417 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_1‚1 /particle/accuracy_3_of_4‚0.9914021492004395 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8879767060279846 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.038219425827264786 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 +jet/accuracy_3_of_4‚0.07302413880825043 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.992394208908081 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3245713412761688 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.06386654078960419 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.05593305081129074 particle/accuracy_2_of_4‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.05314869433641434 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.06427059322595596 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.006776520982384682 6"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7637979388237 +particle/sensitivity‚0.6053935885429382 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9733855724334717 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3478453755378723 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7754084467887878 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4674479067325592 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9125000238418579 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 'particle/f_score‚0.6153185963630676 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6991227865219116 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4687500298023224 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9902834296226501 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.45722484588623047 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.05078916996717453 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5076358914375305 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6200000047683716 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_5‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9669044017791748 particle/accuracy_1_of_4‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6941085457801819 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 % _timestamp‚1.7844873392838886e+09 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6486536860466003 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.48109662532806396 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3210369646549225 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5957640409469604 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.040933139622211456 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7211771011352539 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04172925278544426 )loss/val_total_loss‚13.34665493678025 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9937838315963745 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6200000047683716 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.24940963089466095 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9969725012779236 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.13898107409477234 *jet/accuracy_1_of_2‚0.7302537560462952 jet/accuracy_1_of_5‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3159696161746979 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9772727489471436 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_1_of_4‚0.9536452293395996 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9959161877632141 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9929169416427612 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5627830028533936 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0018424036679789424 particle/accuracy_2_of_2‚1 particle/accuracy_4_of_5‚0 _step‚129 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.04871188476681709 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 -particle/accuracy_4_of_4‚0.94176185131073 _runtime‚ 815.85931326 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3447849452495575 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9772727489471436 /particle/accuracy_3_of_3‚0.9994708299636841 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8478109240531921 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.977369487285614 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04385865852236748 +jet/accuracy_2_of_2‚0.11688835173845291 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.05004241317510605 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9907890558242798 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6437139511108398 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9869071245193481 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9885274171829224 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.050341345369815826 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.08596493303775787 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.712155282497406 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21784032881259918 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5856064558029175 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9719725847244263 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0012339820386841893 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.0616031214594841 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.643651008605957 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6190557479858398 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.41290828585624695 particle/accuracy_1_of_3‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5423077344894409 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9875338673591614 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7055718302726746 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_3_of_5‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7113345861434937 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6508759260177612 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.354938268661499 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9969725012779236 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 (particle/accuracy‚0.8374565839767456 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 *jet/accuracy_1_of_3‚0.9054679870605469 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.1301817148923874 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3453722596168518 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02442612312734127 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0375988632440567 epoch‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6808167695999146 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9978164434432983 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5912607312202454 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48500001430511475 ,jet/accuracy_3_of_3‚0.048241570591926575 +jet/accuracy_2_of_3‚0.45807769894599915 trainer/global_step‚1601 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.17499999701976776 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9876542687416077 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.06254778802394867 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.43426749110221863 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9810134172439575 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5685080885887146 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8490267992019653 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.37121209502220154 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.06163220852613449 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.05078916996717453 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6867520213127136 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.06627216190099716 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.0814136266708374 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.49215081334114075 9 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.051610689610242844 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.07155521959066391 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.809947669506073 6validation_average_jet_accuracy‚0.3017549216747284 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.809947669506073 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.17499999701976776 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 jet/accuracy_2_of_5‚1ÐaãL¯°÷{ 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8490267992019653 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.37121209502220154 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.06163220852613449 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.05078916996717453 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6867520213127136 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.06627216190099716 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.0814136266708374 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.49215081334114075 9 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.051610689610242844 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.07155521959066391 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.809947669506073 6validation_average_jet_accuracy‚0.3017549216747284 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.809947669506073 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.17499999701976776 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 jet/accuracy_2_of_5‚1 8"Purity/2FRt0SRqqt1FBt/event_purity‚0.173611119389534 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9439451098442078 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5957640409469604 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7138746380805969 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.017450423911213875 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7937012910842896 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.14521166682243347 +jet/accuracy_4_of_4‚0.01230158656835556 particle/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4197208285331726 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.012225205078721046 particle/accuracy_2_of_3‚1 *jet/accuracy_2_of_4‚0.7699335813522339 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.11354006081819534 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04215589538216591 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_2‚1 +particle/specificity‚0.9009271264076233 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8364434838294983 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5084346532821655 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.03980943560600281 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8571428656578064 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.06623491644859314 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.2857142984867096 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4199839234352112 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.43201977014541626 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.06354166567325592 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4027476906776428 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6925298571586609 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5875620245933533 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7809703350067139 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7541711926460266 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04385865852236748 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.2857142984867096 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 jet/accuracy_4_of_5‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4328698217868805 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.0626683458685875 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5900972485542297 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1780446618795395 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6000000238418579 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6433731913566589 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5046546459197998 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.14272376894950867 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6514992713928223 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6880733966827393 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5679411888122559 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9885139465332031 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9876542687416077 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4930236339569092 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.1756656914949417 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_1‚1 /particle/accuracy_3_of_4‚0.9914021492004395 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8879767060279846 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.038219425827264786 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 +jet/accuracy_3_of_4‚0.07302413880825043 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.992394208908081 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3245713412761688 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.06386654078960419 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.05593305081129074 particle/accuracy_2_of_4‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.05314869433641434 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.06427059322595596 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.006776520982384682 6"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7637979388237 +particle/sensitivity‚0.6053935885429382 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9733855724334717 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3478453755378723 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7754084467887878 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4674479067325592 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9125000238418579 œ}[–N @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 'particle/f_score‚0.6153185963630676 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6991227865219116 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4687500298023224 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9902834296226501 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.45722484588623047 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.05078916996717453 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5076358914375305 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6200000047683716 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_5‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9669044017791748 particle/accuracy_1_of_4‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6941085457801819 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 % _timestamp‚1.7844873392838886e+09 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6486536860466003 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.48109662532806396 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3210369646549225 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5957640409469604 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.040933139622211456 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7211771011352539 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04172925278544426 )loss/val_total_loss‚13.34665493678025 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9937838315963745 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6200000047683716 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.24940963089466095 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9969725012779236 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.13898107409477234 *jet/accuracy_1_of_2‚0.7302537560462952 jet/accuracy_1_of_5‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3159696161746979 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9772727489471436 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_1_of_4‚0.9536452293395996 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9959161877632141 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9929169416427612 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5627830028533936 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0018424036679789424 particle/accuracy_2_of_2‚1 particle/accuracy_4_of_5‚0 _step‚129 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.04871188476681709 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 -particle/accuracy_4_of_4‚0.94176185131073 _runtime‚ 815.85931326 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3447849452495575 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9772727489471436 /particle/accuracy_3_of_3‚0.9994708299636841 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8478109240531921 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.977369487285614 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04385865852236748 +jet/accuracy_2_of_2‚0.11688835173845291 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.05004241317510605 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9907890558242798 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6437139511108398 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9869071245193481 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9885274171829224 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.050341345369815826 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.08596493303775787 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.712155282497406 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21784032881259918 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5856064558029175 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9719725847244263 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0012339820386841893 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.0616031214594841 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.643651008605957 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6190557479858398 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.41290828585624695 particle/accuracy_1_of_3‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5423077344894409 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9875338673591614 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7055718302726746 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_3_of_5‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7113345861434937 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6508759260177612 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.354938268661499 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9969725012779236 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 (particle/accuracy‚0.8374565839767456 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 *jet/accuracy_1_of_3‚0.9054679870605469 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.1301817148923874 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3453722596168518 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02442612312734127 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0375988632440567 epoch‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6808167695999146 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9978164434432983 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5912607312202454 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48500001430511475 ,jet/accuracy_3_of_3‚0.048241570591926575 +jet/accuracy_2_of_3‚0.45807769894599915 trainer/global_step‚1601 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.17499999701976776 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9876542687416077 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.06254778802394867 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.43426749110221863 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9810134172439575 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5685080885887146 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1¤I°I±‚(:¾ ´»ôÒ¼î€Ì$ memory_percent‚96.98383808135986 cpu‚26.1447086194626 disk.dm-1.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.out‚ 9266.140625 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚3043.49462890625( proc.memory.availableMB‚ 494.16796875 disk.md0.out‚ 25.4609375 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚10971.40576171875 disk.dm-1.out‚1284.126953125 disk.rbd0.out‚0' disk.nvme1c1n1.out‚3043.49462890625 proc.memory.rssMB‚ 2841.5625 proc.cpu.threads‚17 network.recv‚266025% disk./.usageGB‚1176.3009986877441 disk.md0.in‚0 disk.nvme0n1.out‚ 9266.140625 disk.nvme1n1.in‚0* proc.memory.percent‚17.343521118164062 network.sent‚ 1.991059e+06) disk./.usagePercent‚76.61950813794132 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375º»Æž²‚(:“ ´»ôÒŸ­žÍ gpu.0.gpu‚71.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚54.0 gpu.0.powerWatts‚231.598' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚77.19933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0褳E0³+ _wandbruntime‚839  _runtime‚839]MéPÆ´À _runtime‚ 839.431429732 _step‚130 & lr-AdamW/pg1‚0.0002126041947814629 & lr-AdamW/pg2‚0.0002126041947814629 trainer/global_step‚1649 % _timestamp‚1.7844873628546715e+09¦^Ã˵Š% _timestamp‚1.7844873628546715e+09 _runtime‚ 839.431429732 _step‚130 & lr-AdamW/pg1‚0.0002126041947814629 & lr-AdamW/pg2‚0.0002126041947814629 trainer/global_step‚1649‚X„_R0¶+ _wandbruntime‚839  _runtime‚839RÇ‹éà'·ÚO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29553686 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1584602980352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚126586795226445 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚126330405807616 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚101999734224896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚127914991269376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚100365600478720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚126587152189123 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12864610 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1634133746176 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚127915349573632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16678932 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚127914991269376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚125005041440240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚127915349573632 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304256 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚100365624903168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 _step‚131 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241371 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 % _timestamp‚1.7844873628552845e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1584943766016 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚126330388289024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4358701 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 trainer/global_step‚1649 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚126330388289024 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚126330405807616 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚101999883431424 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16678932 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494208 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12864610 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1584602980352 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16689074 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8757682 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 trainer/global_step‚1649 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚126330388289024 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚126330405807616 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚101999883431424 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16678932 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494208 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12864610 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1584602980352 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16689074 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8757682 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1584602980352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚126586795226445 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚126330405807616 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚101999734224896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚127914991269376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚100365600478720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚126587152189123 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12864610 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1634133746176 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚127915349573632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16678932 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚127914991269376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚125005041440240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚127915349573632 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304256 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚100365624903168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 _step‚131 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241371 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 % _timestamp‚1.7844873628552845e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1584943766016 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚126330388289024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4358701 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1585873684480 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.oZëäóRn_train_batch_end/allocated_bytes.small_pool.allocated‚ 1585873684480 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12872112 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4360928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125078467522720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29571077 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚126404580058112 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13123689 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29560919 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚126404580058112 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792832 _runtime‚ 839.789004669 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16688807 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚126404597576704 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚127990471261184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚100424476811264 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1583021932871 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29571077 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1585532891648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16698963 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8762761 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12872114 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 _step‚132 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16698963 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001083 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1635088130048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚127990112949760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 % _timestamp‚1.7844873632138116e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102059564941312 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚125078484562080 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962790 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29560919 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1635225487872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8762763 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1582682009441 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 137357824 trainer/global_step‚1649 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720éLÆÍï&»éM E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1635088130048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚127990112949760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 % _timestamp‚1.7844873632138116e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102059564941312 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚125078484562080 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962790 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29560919 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1635225487872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8762763 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1582682009441 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 137357824 trainer/global_step‚1649 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12872114 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311424 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1585873684480 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1585873684480 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12872112 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4360928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125078467522720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29571077 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚126404580058112 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13123689 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29560919 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚126404580058112 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792832 _runtime‚ 839.789004669 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16688807 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚126404597576704 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚127990471261184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚100424476811264 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1583021932871 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29571077 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1585532891648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16698963 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8762761 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12872114 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 _step‚132 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16698963 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001083 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368„8Ň}0¼+ _wandbruntime‚839  _runtime‚839Né(½ý 0loss/FBt2/assignment_loss‚0.6594023504959685 1loss/SRqqt2/detection_loss‚1.1162732202954442 .loss/FBt2/detection_loss‚4.530972744737427 _step‚133 0loss/FRt1/assignment_loss‚1.5687213140765046 1loss/SRqqt2/assignment_loss‚1.025849611609434 epoch‚2 /loss/FRt2/assignment_loss‚3.047681699673682 2loss/SRqqt1/assignment_loss‚0.7530064851189019 1loss/FBt1/assignment_loss‚0.04226519887500656 /loss/FRt2/detection_loss‚0.5697736864533086 1loss/SRqqt1/detection_loss‚0.6876666265535433 /loss/FBt1/detection_loss‚0.8638034081750475 _runtime‚ 839.789862651 /loss/FRt1/detection_loss‚1.3877826431915286 &loss/total_loss‚16.253198989255797 trainer/global_step‚1649 % _timestamp‚1.7844873632151916e+09zðíñ¾‚ 0loss/FRt1/assignment_loss‚1.5687213140765046 1loss/SRqqt2/assignment_loss‚1.025849611609434 epoch‚2 /loss/FRt2/assignment_loss‚3.047681699673682 2loss/SRqqt1/assignment_loss‚0.7530064851189019 1loss/FBt1/assignment_loss‚0.04226519887500656 /loss/FRt2/detection_loss‚0.5697736864533086 1loss/SRqqt1/detection_loss‚0.6876666265535433 /loss/FBt1/detection_loss‚0.8638034081750475 _runtime‚ 839.789862651 /loss/FRt1/detection_loss‚1.3877826431915286 &loss/total_loss‚16.253198989255797 trainer/global_step‚1649 % _timestamp‚1.7844873632151916e+09 0loss/FBt2/assignment_loss‚0.6594023504959685 1loss/SRqqt2/detection_loss‚1.1162732202954442 .loss/FBt2/detection_loss‚4.530972744737427 _step‚133…[ÍS¿‚(:È ûôÒ÷ªÑË network.recv‚270083 disk.nvme0n1.in‚ 4.3359375' disk.nvme1c1n1.out‚3068.24853515625 disk.md0.out‚ 25.81640625 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚10973.54345703125 disk.dm-1.in‚0 network.sent‚ 2.035413e+06 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0% disk.nvme1n1.out‚3068.24853515625 disk.md0.in‚0 proc.memory.rssMB‚ 2841.5625) disk./.usagePercent‚76.61951062268145 disk.rbd0.in‚0& disk.nvme0c0n1.out‚9268.4423828125$ disk.nvme0n1.out‚9268.4423828125$ memory_percent‚96.98548316955566 cpu‚26.137798931743774 proc.cpu.threads‚17% disk./.usageGB‚1176.3010368347168 disk.nvme0c0n1.in‚ 4.3359375 disk.dm-1.out‚1308.271484375' proc.memory.availableMB‚ 493.8984375* proc.memory.percent‚17.343521118164062[íÿQ‘À‚(:† ûôÒÀ¢³Ì gpu.0.gpu‚56.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚118.74' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.58 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05“r~QÁ‚(:Æ Ò»ôÒˆ¶ÔË proc.cpu.threads‚17% disk./.usageGB‚1176.3014602661133 disk.dm-0.in‚ 4.3359375 disk.dm-1.out‚1529.673828125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.3359375% disk.nvme1n1.out‚3290.31201171875$ memory_percent‚96.98412418365479 proc.memory.rssMB‚ 2841.5625 network.sent‚ 2.098826e+06 network.recv‚274164 disk.md0.in‚0# disk.dm-0.out‚10974.48876953125% disk.nvme0c0n1.out‚9269.884765625 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0( proc.memory.availableMB‚ 494.12109375 disk.dm-1.in‚0 disk.rbd0.in‚0' disk.nvme1c1n1.out‚3290.31201171875 disk.nvme1n1.in‚0 cpu‚26.131678261484915) disk./.usagePercent‚76.61953820329674 disk.md0.out‚ 26.3671875# disk.nvme0n1.out‚9269.884765625* proc.memory.percent‚17.3435211181640621¹Ù÷žÂ‚(:“ Ò»ôÒ­–šÌ gpu.0.gpu‚70.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚264.584' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.19466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³-vi0Ã+ _wandbruntime‚863  _runtime‚863á)šCÈÄ % _timestamp‚1.7844873867854953e+09 _runtime‚ 863.360476896 _step‚134 ' lr-AdamW/pg1‚0.00021257445654777364 ' lr-AdamW/pg2‚0.00021257445654777364 trainer/global_step‚1699Ì›¨ÍÅÇ _runtime‚ 863.360476896 _step‚134 ' lr-AdamW/pg1‚0.00021257445654777364 ' lr-AdamW/pg2‚0.00021257445654777364 trainer/global_step‚1699 % _timestamp‚1.7844873867854953e+09† š~0Æ+ _wandbruntime‚863  _runtime‚863êYÒà'ÇÚO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17184676 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚130039976743424 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚131671465849856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001083 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494208 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30424388 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962678 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚104991616939520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚1699 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚104991461441536 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1628536774531 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13239712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4484123 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1682175409152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7844873867863464e+09 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4484402 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131671107545600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13495853 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304256 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1631130802176 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚131671107545600 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13239710 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9011732 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785664 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚130304899378131 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1631471587840 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131671107545600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13495853 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304256 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1631130802176 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚131671107545600 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13239710 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9011732 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785664 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚130304899378131 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1631471587840 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001083 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494208 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30424388 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962678 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚104991616939520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚1699 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚104991461441536 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1628536774531 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13239712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4484123 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1682175409152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7844873867863464e+09 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4484402 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311424 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1632060713472 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128969216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1683003730944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13247214 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4486652 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚130114168512512 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚103368293438464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1683132700160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1629464997879 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001083 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 trainer/global_step‚1699 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494208 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚130114186031104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30431621 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 863.718570242 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9016813 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9016811 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128749788686080 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30441779 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17184409 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17194565 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17184409 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13503737 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4486924 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962790 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30441779 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17194565 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13503463 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚105051297169408 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1632401506304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1632060713472 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13247214 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7844873871434212e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚130379253683959 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311424 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1632060713472 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128969216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1683003730944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13247214 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4486652 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚130114168512512 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚103368293438464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1683132700160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1629464997879 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001083 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 trainer/global_step‚1699 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494208 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚130114186031104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30431621 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 863.718570242 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9016813 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9016811 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128749788686080 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30441779 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17184409 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17194565 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17184409 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13503737 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4486924 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962790 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30441779 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17194565 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13503463 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚105051297169408 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1632401506304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1632060713472 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13247214 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7844873871434212e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚130379253683959 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259ˆcLBQ0Ì+ _wandbruntime‚863  _runtime‚863ŽÂšÍý /loss/FBt1/detection_loss‚0.8104176202599664 /loss/FBt2/detection_loss‚1.4893519353298914 % _timestamp‚1.7844873871443882e+09 _step‚137 2loss/SRqqt2/assignment_loss‚0.8780815411214782 1loss/SRqqt1/detection_loss‚0.7356002428677424 trainer/global_step‚1699 /loss/FRt1/assignment_loss‚1.330647386361232 2loss/SRqqt1/assignment_loss‚0.7249438104855129 1loss/SRqqt2/detection_loss‚1.2669699883193408 epoch‚2 /loss/FRt2/assignment_loss‚2.905239913851843 0loss/FBt1/assignment_loss‚0.2338411953971997 /loss/FRt1/detection_loss‚1.4301074500469193 &loss/total_loss‚12.532515857719591 _runtime‚ 863.719019956 0loss/FBt2/assignment_loss‚0.1164207533120134 /loss/FRt2/detection_loss‚0.6108940203664526ýÌ*â΂ /loss/FBt2/detection_loss‚1.4893519353298914 % _timestamp‚1.7844873871443882e+09 _step‚137 2loss/SRqqt2/assignment_loss‚0.8780815411214782 1loss/SRqqt1/detection_loss‚0.7356002428677424 trainer/global_step‚1699 /loss/FRt1/assignment_loss‚1.330647386361232 2loss/SRqqt1/assignment_loss‚0.7249438104855129 1loss/SRqqt2/detection_loss‚1.2669699883193408 epoch‚2 /loss/FRt2/assignment_loss‚2.905239913851843 0loss/FBt1/assignment_loss‚0.2338411953971997 /loss/FRt1/detection_loss‚1.4301074500469193 &loss/total_loss‚12.532515857719591 _runtime‚ 863.719019956 0loss/FBt2/assignment_loss‚0.1164207533120134 /loss/FRt2/detection_loss‚0.6108940203664526 /loss/FBt1/detection_loss‚0.8104176202599664‰0ârAPÏ‚(:Å á»ôÒÁâÊË disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1n1.in‚0 disk.dm-0.in‚ 4.3359375 proc.cpu.threads‚17 network.sent‚ 2.102683e+06 disk.rbd0.out‚0% disk.nvme1c1n1.out‚3291.154296875# disk.nvme1n1.out‚3291.154296875 disk.md0.in‚0( proc.memory.availableMB‚ 494.08984375* proc.memory.percent‚17.343521118164062 cpu‚26.12577153840203% disk./.usageGB‚1176.2548637390137 disk.nvme0n1.in‚ 4.3359375 disk.md0.out‚ 26.8515625" disk.dm-1.out‚1529.82763671875$ memory_percent‚96.98431491851807& disk.nvme0c0n1.out‚9272.0166015625$ disk.nvme0n1.out‚9272.0166015625 disk.nvme1c1n1.in‚0# disk.dm-0.out‚10976.28369140625 disk.dm-1.in‚0 proc.memory.rssMB‚ 2841.5625 network.recv‚277520) disk./.usagePercent‚76.61650309324304Aþ9»Ð‚(:” á»ôÒÅó¬Ì gpu.0.gpu‚91.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.ã?jäpowerWatts‚166.379' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚55.459666666666664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÔjCÈUÑ‚(:Ê ð»ôÒõîË network.sent‚ 2.16504e+06 disk.rbd0.in‚0% disk.nvme1n1.out‚3291.91748046875 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0$ memory_percent‚96.98450565338135 proc.cpu.threads‚17 network.recv‚281954' disk.nvme0c0n1.out‚9273.20947265625 disk.nvme0n1.in‚ 4.3359375 disk.md0.in‚0 disk.md0.out‚ 27.41796875" disk.dm-1.out‚1529.93310546875( proc.memory.availableMB‚ 494.05859375* proc.memory.percent‚17.343521118164062 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0# disk.dm-0.out‚10976.95947265625 cpu‚26.1200399803723' disk.nvme1c1n1.out‚3291.91748046875) disk./.usagePercent‚76.61650682035321% disk./.usageGB‚1176.2549209594727 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.3359375% disk.nvme0n1.out‚9273.20947265625 proc.memory.rssMB‚ 2841.5625Þ¨6‘Ò‚(:† ð»ôÒð͵Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚54.0 gpu.0.powerWatts‚118.557' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.519 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´+-!0Ó+ _wandbruntime‚887  _runtime‚887^“hâÈÔ trainer/global_step‚1749 % _timestamp‚1.7844874106958907e+09 _runtime‚ 887.270862388 _step‚138 ' lr-AdamW/pg1‚0.00021254301810914582 ' lr-AdamW/pg2‚0.00021254301810914582Ö¾\ÍÕÇ % _timestamp‚1.7844874106958907e+09 _runtime‚ 887.270862388 _step‚138 ' lr-AdamW/pg1‚0.00021254301810914582 ' lr-AdamW/pg2‚0.00021254301810914582 trainer/global_step‚1749Š Á60Ö+ _wandbruntime‚887  _runtime‚887¥y§à'×ÚO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13614812 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13876067 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚132347649688240 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚135427582126080 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚132347666727600 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 trainer/global_step‚1749 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31284946 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚106253258157568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚134022646567139 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4610287 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1677999409664 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31295090 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13614812 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9265782 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 135267840 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 887.271573849 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17670136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚106253233733120 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135427223821824 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚107983220690432 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1677658624000 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17680278 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚135427223821824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚107983380382720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1730122225152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304256 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚133749565197824 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1677999409664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592Á$~å'ØßO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚133749582716416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1674639916221 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7844874106967409e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚135427223821824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚107983380382720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1730122225152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304256 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚133749565197824 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1677999409664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13614812 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13876067 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚132347649688240 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚135427582126080 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚132347666727600 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 trainer/global_step‚1749 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31284946 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚106253258157568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚134022646567139 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4610287 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1677999409664 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31295090 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13614812 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9265782 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 135267840 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 887.271573849 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17670136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚106253233733120 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135427223821824 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚107983220690432 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1677658624000 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17680278 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13622312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9270861 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚135502703813632 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚133823756966912 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 % _timestamp‚1.7844874110550327e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311424 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17680011 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31312481 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚134097000872967 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 _step‚140 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4612818 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13622314 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13622312 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚135502703813632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1678588535296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31302323 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13883969 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 137357824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚133823774485504 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13622314 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108043056922112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870272 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚134096643910177 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31312481 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1678929328128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962790 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚133823774485504 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚106312110065664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1678588535296 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1730946856448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108043218704384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚106312134490112 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17680011 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1678929328128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792832 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17690167 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792832 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1675908062887 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4613106 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13883679 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1731084214272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9270863 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720ýBøÛéM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13883679 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1731084214272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMoIãÜá%nitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9270863 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚132421092810080 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚132421075770720 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001083 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13622312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9270861 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚135502703813632 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚133823756966912 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 % _timestamp‚1.7844874110550327e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311424 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17680011 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31312481 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚134097000872967 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 _step‚140 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4612818 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13622314 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13622312 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚135502703813632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1678588535296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31302323 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13883969 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 137357824 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚133823774485504 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13622314 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108043056922112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870272 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚134096643910177 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31312481 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1678929328128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962790 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚133823774485504 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚106312110065664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1678588535296 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1730946856448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108043218704384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚106312134490112 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17680011 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1678929328128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792832 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17690167 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792832 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1675908062887 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4613106 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4737746 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494208 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494208 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13989912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001083 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 _step‚143 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32155648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13989910 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923318 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18175880 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚137459153652224 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139183698402304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1778076948480 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139183340098048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1724186445824 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1721422904547 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14257291 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18175880 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784487434632436e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚137740036793469 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1777952166400 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚136018970851600 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18165738 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚136018953812240 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚110975151733248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304256 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1724186445824 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚137740393756147 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13989912 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚137459153652224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 trainer/global_step‚1799 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32165792 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32165792 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4737746 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494208 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494208 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13989912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001083 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 _step‚143 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32155648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13989910 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923318 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18175880 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚137459153652224 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139183698402304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1778076948480 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139183340098048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1724186445824 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1721422904547 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14257291 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18175880 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784487434632436e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚137740036793469 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1777952166400 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚136018970851600 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18165738 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚136018953812240 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚110975151733248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.la e rge_pool.peak‚351 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304256 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1724186445824 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚137740393756147 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13989912lêÿ0ç+ _wandbruntime‚911  _runtime‚911æµEré&èãM EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚137533362939904 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚139258820089856 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137533362939904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1725116357120 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 % _timestamp‚1.7844874349896142e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9524913 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4740000 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1778908295168 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13997412 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774912 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18185769 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14264911 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13997412 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1722351127895 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32183183 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32183183 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚137533345421312 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149199360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚109255926692864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚137814391099185 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1722011204465 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792832 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9524911 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1779033070080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚136092379894720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494208 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32173025 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚111034984187392 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001083 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1725116357120 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1725457149952 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494208 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32173025 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚111034984187392 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001083 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1725116357120 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1725457149952 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚137533362939904 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚139258820089856 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137533362939904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1725116357120 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 % _timestamp‚1.7844874349896142e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9524913 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4740000 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1778908295168 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13997412 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774912 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18185769 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14264911 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13997412 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1722351127895 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32183183 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32183183 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚137533345421312 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149199360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚109255926692864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚137814391099185 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1722011204465 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792832 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9524911 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1779033070080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚136092379894720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚142939814678528 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206528 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112140866987520 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14365012 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚141168742106624 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚142939456374272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚142939456374272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚141168742106624 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9773882 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9773880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124782080 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_trai¥;œÁ[ n_batch_start/segment.large_pool.current‚675 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1771055053312 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14365010 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18661340 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14365012 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 _step‚147 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1825883190272 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962678 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚1849 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14638022 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1771055053312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785664 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4864412 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚141458140945155 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304256 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18671482 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚139690257936240 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1770714267648 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚141168759625216 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870272 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚142939814678528 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206528 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112140866987520 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14365012 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚141168742106624 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚142939456374272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚142939456374272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚141168742106624 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9773882 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9773880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124782080 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1771055053312 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14365010 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18661340 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14365012 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 _step‚147 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1825883190272 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962678 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚1849 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14638022 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1771055053312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785664 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4864412 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚141458140945155 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304256 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18671482 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚139690257936240 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1770714267648 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚141168759625216 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870272 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚141532495250983 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚139763701058080 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923430 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33053885 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14372514 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚114026738394112 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚112199767744512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33043727 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112199743320064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚143014936366080 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚114026591291904 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14645546 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 trainer/global_step‚1849 _runtime‚ 935.515652319 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14372514 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14372512 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18681371 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18681371 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚139763684018720 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4866585 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18671215 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122677760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33043727 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚141242951394304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18671215 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚143014936366080 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 % _timestamp‚1.7844874589405277e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33053885 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1826847971840 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1771984971776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675§×üA…%ùéM PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 trainer/global_step‚1849 _runtime‚ 935.515652319 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14372514 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14372512 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18681371 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18681371 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚139763684018720 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4866585 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18671215 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122677760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33043727 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚141242951394304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18671215 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚143014936366080 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 % _timestamp‚1.7844874589405277e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33053885 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1826847971840 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1771984971776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311424 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚141532495250983 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚139763701058080 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923430 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33053885 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14372514 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚114026738394112 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚112199767744512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33043727 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112199743320064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚143014936366080 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚114026591291904 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14645546”–é"0ú+ _wandbruntime‚935  _runtime‚935µ¬öHûü /loss/FBt2/detection_loss‚3.8061449420120983 % _timestamp‚1.7844874589415095e+09 .loss/FRt2/assignment_loss‚2.77878273513403 1loss/FBt1/assignment_loss‚0.04580077626618784 &loss/total_loss‚14.536838708474605 epoch‚2 _step‚149 .loss/FRt1/detection_loss‚1.471094651630075 1loss/SRqqt2/detection_loss‚0.9983203742944495 trainer/global_step‚1849 _runtime‚ 935.51615963 2loss/SRqqt2/assignment_loss‚0.8553885605092572 1loss/FBt2/assignment_loss‚0.07328451867700399 /loss/FRt2/detection_loss‚0.6323959071826041 /loss/FBt1/detection_loss‚1.0162193972658844 0loss/FRt1/assignment_loss‚1.3723390950524252 1loss/SRqqt1/assignment_loss‚0.778550497864273 1loss/SRqqt1/detection_loss‚0.7085172525863153d·wü 1loss/SRqqt1/assignment_loss‚0.778550497864273 1loss/SRqqt1/detection_loss‚0.7085172525863153 /loss/FBt2/detection_loss‚3.8061449420120983 % _timestamp‚1.7844874589415095e+09 .loss/FRt2/assignment_loss‚2.77878273513403 1loss/FBt1/assignment_loss‚0.04580077626618784 &loss/total_loss‚14.536838708474605 epoch‚2 _step‚149 .loss/FRt1/detection_loss‚1.471094651630075 1loss/SRqqt2/detection_loss‚0.9983203742944495 trainer/global_step‚1849 _runtime‚ 935.51615963 2loss/SRqqt2/assignment_loss‚0.8553885605092572 1loss/FBt2/assignment_loss‚0.07328451867700399 /loss/FRt2/detection_loss‚0.6323959071826041 /loss/FBt1/detection_loss‚1.0162193972658844 0loss/FRt1/assignment_loss‚1.3723390950524252•âß*eHý‚(:½ ¬¼ôÒÊð’Ì proc.memory.rssMB‚ 2841.5625 network.sent‚ 2.312411e+06 disk.nvme0n1.in‚ 4.3359375! disk.nvme0n1.out‚ 9279.6640625 disk.md0.out‚ 29.33203125# disk.nvme0c0n1.out‚ 9279.6640625( proc.memory.availableMB‚ 493.65234375 network.recv‚295914$ disk./.usageGB‚1176.379623413086 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0# disk.nvme1n1.out‚3300.681640625 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375* proc.memory.percent‚17.343521118164062% disk.nvme1c1n1.out‚3300.681640625 disk.dm-1.in‚0" disk.dm-1.out‚1530.40478515625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.3359375 cpu‚26.09817370890858 proc.cpu.threads‚17) disk./.usagePercent‚76.62462943579781# disk.dm-0.out‚10987.61376953125" memory_percent‚96.986985206604­Oþ‚(:’ ¬¼ôÒ£ðÄÌ gpu.0.gpu‚70.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚54.0 gpu.0.powerWatts‚268.64' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.54666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷®0ÿ+ _wandbruntime‚958  _runtime‚958@ mÅ€ ¿ & lr-AdamW/pg1‚0.0002124385067352473 & lr-AdamW/pg2‚0.0002124385067352473 trainer/global_step‚1899 $ _timestamp‚1.784487482197683e+09 _runtime‚ 958.772661695 _step‚150Å¿}Ê Ä trainer/global_step‚1899 $ _timestamp‚1.784487482197683e+09 _runtime‚ 958.772661695 _step‚150 & lr-AdamW/pg1‚0.0002124385067352473 & lr-AdamW/pg2‚0.0002124385067352473–¬éÜ0‚ + _wandbruntime‚958  _runtime‚958)O|â'ƒ ÜO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14740112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33897052 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚144878330561024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870272 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33907196 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001083 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923318 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15018315 _runtime‚ 958.773391246 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚146695930954752 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1873973528064 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1817582875136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1817242089472 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4990383 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1873855037440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚146695572650496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚115084708039168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494208 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10027930 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870272 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚143361579099600 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚116958681567232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962678 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1814309034563 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14740110 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚144878348079616 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚144878330561024 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚116958538652160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚144878348079616 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚1899 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 142915072 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1813969111245 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15018052 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚146695930954752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚143361562060240 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10027932 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19167084 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1817582875136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚145175531171485 % _timestamp‚1.7844874821985412e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241371 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚115084683614720 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚146695572650496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1817242089472 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19167084 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675¾å¼…ç'„ áO @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15018052 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚146695930954752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚143361562060240 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10027932 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19167084 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1817582875136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚145175531171485 % _timestamp‚1.7844874821985412e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241371 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚115084683614720 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚146695572650496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1817242089472 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19167084 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14740112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33897052 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚144878330561024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870272 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33907196 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001083 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923318 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15018315 _runtime‚ 958.773391246 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚146695930954752 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1873973528064 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1817582875136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1817242089472 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4990383 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1873855037440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚146695572650496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚115084708039168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494208 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10027930 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870272 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚143361579099600 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚116958681567232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962678 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1814309034563 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14740110 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚144878348079616 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚144878330561024 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚116958538652160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚144878348079616 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚1899 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 142915072 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1813969111245 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088—¡*¹0… + _wandbruntime‚959  _runtime‚959­eõ.!† åM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1814897334481 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14747612 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚143434988142720 trainer/global_step‚1899 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19176973 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311424 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792832 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14747612 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1874939032576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19166817 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1818172000768 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚145250242439991 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚144952539848704 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚144952539848704 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117018370010624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15025610 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1818172000768 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚146770694330880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14747614 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393664 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33914429 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4992868 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚145249885477201 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10033013 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33924587 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4992599 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143435005182080 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14747614 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792832 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15025881 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚144952522330112 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1874810063360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001083 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 _runtime‚ 959.131425581 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19166817 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923430 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.curren"Ž#½t‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚146771052642304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10033011 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1815237257911 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15025610 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1818172000768 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚146770694330880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14747614 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393664 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33914429 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4992868 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚145249885477201 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10033013 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33924587 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4992599 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143435005182080 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14747614 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792832 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15025881 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚144952522330112 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1874810063360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001083 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 _runtime‚ 959.131425581 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19166817 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923430 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚146771052642304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10033011 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1815237257911 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14747612 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚143434988142720 trainer/global_step‚1899 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19176973 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311424 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792832 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14747612 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1874939032576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19166817 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1818172000768 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚145250242439991 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720˜»]ÔS0ˆ + _wandbruntime‚959  _runtime‚959”:"‰ û 1loss/SRqqt1/detection_loss‚0.6673486203279647 trainer/global_step‚1899 0loss/SRqqt2/detection_loss‚1.104361998205561 .loss/FBt1/detection_loss‚3.399895568401603 _step‚153 0loss/FBt2/assignment_loss‚0.5581525641085752 .loss/FBt2/detection_loss‚1.015750278444851 epoch‚2 $ _timestamp‚1.784487482557238e+09 _runtime‚ 959.131912982 /loss/FRt1/assignment_loss‚1.391984026947301 1loss/FBt1/assignment_loss‚0.07709738069569279 /loss/FRt1/detection_loss‚1.4717172535781105 &loss/total_loss‚14.779527289898024 0loss/FRt2/assignment_loss‚2.9204792585022763 2loss/SRqqt1/assignment_loss‚0.7478941412339315 2loss/SRqqt2/assignment_loss‚0.8337982736550094 /loss/FRt2/detection_loss‚0.5910479257971483]'Š € 0loss/FBt2/assignment_loss‚0.5581525641085752 .loss/FBt2/detection_loss‚1.015750278444851 epoch‚2 $ _timestamp‚1.784487482557238e+09 _runtime‚ 959.131912982 /loss/FRt1/assignment_loss‚1.391984026947301 1loss/FBt1/assignment_loss‚0.07709738069569279 /loss/FRt1/detection_loss‚1.4717172535781105 &loss/total_loss‚14.779527289898024 0loss/FRt2/assignment_loss‚2.9204792585022763 2loss/SRqqt1/assignment_loss‚0.7478941412339315 2loss/SRqqt2/assignment_loss‚0.8337982736550094 /loss/FRt2/detection_loss‚0.5910479257971483 1loss/SRqqt1/detection_loss‚0.6673486203279647 trainer/global_step‚1899 0loss/SRqqt2/detection_loss‚1.104361998205561 .loss/FBt1/detection_loss‚3.399895568401603 _step‚153™{Š8\K‹ ‚(:À »¼ôÒ·ÂØË disk.rbd0.in‚0 disk.md0.in‚0 proc.memory.rssMB‚ 2841.5625) disk./.usagePercent‚76.62848624940844 disk.nvme0n1.in‚ 4.3359375" disk.nvme1c1n1.out‚ 3301.390625 disk.nvme1n1.in‚0 disk.dm-0.in‚ 4.3359375( proc.memory.availableMB‚ 493.62109375* proc.memory.percent‚17.343521118164062 cpu‚26.092836798620397 network.recv‚299715 disk.rbd0.out‚0& disk.nvme0c0n1.out‚9301.5869140625 disk.nvme1n1.out‚ 3301.390625 disk.md0.out‚ 29.58984375# disk.dm-0.out‚11009.52685546875$ disk./.usageGB‚1176.438835144043 disk.nvme0c0n1.in‚ 4.3359375$ disk.nvme0n1.out‚9301.5869140625 disk.nvme1c1n1.in‚0$ memory_percent‚96.98717594146729 proc.cpu.threads‚17 network.sent‚ 2.360857e+06 disk.dm-1.in‚0" disk.dm-1.out‚1530.55712890625pFMÿŒ ‚(:’ »¼ôÒä´²Ì gpu.0.gpu‚65.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚54.0 gpu.0.powerWatts‚240.61' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚80.20333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M­àS ‚(:È ʼôÒæ™Í' disk.nvme1c1n1.out‚3301.80517578125 proc.memory.rssMB‚ 2841.5625 proc.cpu.threads‚17 network.sent‚ 2.425203e+06) disk./.usagePercent‚76.62851631476386 disk.nvme1n1.in‚0 disk.md0.in‚0# disk.dm-0.out‚11051.08154296875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.3359375% disk.nvme1n1.out‚3301.80517578125 disk.dm-0.in‚ 4.3359375 disk.nvme1c1n1.in‚0* proc.memory.percent‚17.343521118164062 cpu‚26.088069189786683 disk.md0.out‚ 29.7890625" disk.dm-1.out‚1530.65478515625' disk.nvme0c0n1.out‚9343.27099609375$ memory_percent‚96.98734283447266% proc.memory.availableMB‚ 493.59375 network.recv‚304259$ disk./.usageGB‚1176.439296722412 disk.dm-1.in‚0 disk.nvme0n1.in‚ 4.3359375% disk.nvme0n1.out‚9343.27099609375¤¿B®žŽ ‚(:“ ʼôÒ°«¶Í gpu.0.gpu‚64.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚253.168' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.38933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…Q¨0 + _wandbruntime‚982  _runtime‚982u¿ÐãÄ ¾ % lr-AdamW/pg1‚0.000212400272784568 % lr-AdamW/pg2‚0.000212400272784568 trainer/global_step‚1949 % _timestamp‚1.7844875061608522e+09 _runtime‚ 982.735819148 _step‚154U"É‘ à % lr-AdamW/pg2‚0.000212400272784568 trainer/global_step‚1949 % _timestamp‚1.7844875061608522e+09 _runtime‚ 982.735819148 _step‚154 % lr-AdamW/pg1‚0.000212400272784568šŠóc0’ + _wandbruntime‚982  _runtime‚982-„•á'“ ÛO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19652544 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚148587936534016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚118028500241920 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870272 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚148587936534016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚ 982.736601889 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1863769911296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5117238 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494208 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15398936 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚119950539180032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1864110696960 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34777898 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10281980 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1860752099571 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚147032866184240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10281982 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚148587919015424 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚118028524666368 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚148893278360493 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241371 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304256 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19652544 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962678 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150451688926720 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870272 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚119950402556416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923318 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 _step‚155 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 $ _timestamp‚1.784487506161752e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15115212 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚148587919015424 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1921902314496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001083 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19662686 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19662686 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15115210 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1860412176253 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675Ðë¤ ” àO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚1949 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15115210 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785664 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚150451688926720 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5116956 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 136623616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15399220 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 112199168 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1863769911296 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚147032883223600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚150452047230976 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34767754 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚150452047230976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚148893635323171 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304256 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150451688926720 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870272 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚119950402556416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923318 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 _step‚155 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 $ _timestamp‚1.784487506161752e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15115212 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚148587919015424 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1921902314496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001083 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19662686 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19662686 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15115210 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1860412176253 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19652544 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚148587936534016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚118028500241920 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870272 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚148587936534016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚ 982.736601889 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1863769911296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5117238 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494208 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15398936 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚119950539180032‚b2Ú ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1864110696960 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34777898 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10281980 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1860752099571 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚147032866184240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10281982 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚148587919015424 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚118028524666368 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚148893278360493 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241371 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304256 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19652544 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962678 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 I 0• + _wandbruntime‚983  _runtime‚983gUtì&– æM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1864699822592 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚118087400998912 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1922982693888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15122714 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001083 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10287063 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34785131 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19672575 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚148967989628999 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15122714 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1864699822592 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚150526810607104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962790 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792832 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚150527168918528 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15406539 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34795289 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 983.097595246 _step‚156 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚148662110784512 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚150527168918528 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19662419 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚120010242882048 % _timestamp‚1.7844875065225036e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚148662110784512 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚120010383692800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5119478 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923430 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚148662128303104 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5119752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 trainer/global_step‚1949 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚148967632666209 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15406815 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19662419 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792832 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34785131 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15122712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19672575 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 140810752 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34795289 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1922866307584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1865040615424 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15122712 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 116386304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34795289 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1922866307584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1865040615424 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15122712 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 116386304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34785131 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19672575 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚148967989628999 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15122714 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1864699822592 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚150526810607104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962790 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792832 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚150527168918528 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15406539 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34795289 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 983.097595246 _step‚156 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚148662110784512 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚150527168918528 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19662419 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚120010242882048 % _timestamp‚1.7844875065225036e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚148662110784512 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚120010383692800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5119478 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923430 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚148662128303104 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5119752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 trainer/global_step‚1949 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚148967632666209 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15406815 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19662419 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792832 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34785131 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15122712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19672575 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156œ$~Îì0˜ + _wandbruntime‚983  _runtime‚983=VÂÙ™ € 2loss/SRqqt1/assignment_loss‚0.6709894039215416 1loss/SRqqt1/detection_loss‚0.6387180081651336 1loss/SRqqt2/detection_loss‚1.1233877574952391 /loss/FBt1/detection_loss‚0.9230769155275422 trainer/global_step‚1949 /loss/FRt2/assignment_loss‚2.858740784478991 % _timestamp‚1.7844875065234923e+09 0loss/FRt1/assignment_loss‚1.2243228300814164 /loss/FRt2/detection_loss‚0.5957863198373509 /loss/FBt2/detection_loss‚1.2659511003229353 &loss/total_loss‚11.885178644437808 _step‚157 2loss/SRqqt2/assignment_loss‚0.7797151924467755 2loss/FBt1/assignment_loss‚0.061894226090721816 1loss/FBt2/assignment_loss‚0.36080050329981517 .loss/FRt1/detection_loss‚1.381795602770344 epoch‚2 _runtime‚ 983.098583188uŒ š … 2loss/SRqqt2/assignment_loss‚0.7797151924467755 2loss/FBt1/assignment_loss‚0.061894226090721816 1loss/FBt2/assignment_loss‚0.36080050329981517 .loss/FRt1/detection_loss‚1.381795602770344 epoch‚2 _runtime‚ 983.098583188 2loss/SRqqt1/assignment_loss‚0.6709894039215416 1loss/SRqqt1/detection_loss‚0.6387180081651336 1loss/SRqqt2/detection_loss‚1.1233877574952391 /loss/FBt1/detection_loss‚0.9230769155275422 trainer/global_step‚1949 /loss/FRt2/assignment_loss‚2.858740784478991 % _timestamp‚1.7844875065234923e+09 0loss/FRt1/assignment_loss‚1.2243228300814164 /loss/FRt2/detection_loss‚0.5957863198373509 /loss/FBt2/detection_loss‚1.2659511003229353 &loss/total_loss‚11.885178644437808 _step‚157 ÎéR› ‚(:Ç Ù¼ôÒ†ƒÄË disk.md0.out‚30.21875 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚11059.80029296875 disk.dm-1.in‚0% disk./.usageGB‚1176.4391860961914 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1n1.in‚0 disk.md0.in‚0! disk.dm-1.out‚1530.7744140625$ memory_percent‚98.49193096160889( proc.memory.availableMB‚ 247.08203125* proc.memory.percent‚17.343521118164062 network.recv‚307441) disk./.usagePercent‚76.62850910901751& disk.nvme0c0n1.out‚9346.4013671875 proc.memory.rssMB‚ 2841.5625 proc.cpu.threads‚17 disk.nvme0n1.in‚ 4.3359375$ disk.nvme0n1.out‚9346.4013671875 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚3308.42236328125 cpu‚26.083098494800105 network.sent‚ 2.428167e+06 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme1n1.out‚3308.42236328125°Ú¡ç“œ ‚(:ˆ Ù¼ôÒìü¤Ì gpu.0.gpu‚91.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚119.121' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.707 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓÒa*T ‚(:É è¼ôÒŸìõË$ disk./.usageGB‚1176.439640045166 disk.md0.out‚ 30.76953125 network.recv‚311851) disk./.usagePercent‚76.62853867742491 disk.dm-0.in‚ 4.3359375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375' disk.nvme0c0n1.out‚9348.24951171875 disk.nvme0n1.in‚ 4.3359375( proc.memory.availableMB‚ 247.07421875 network.sent‚ 2.492336e+06' disk.nvme1c1n1.out‚3309.19677734375 disk.dm-1.in‚0! disk.dm-1.out‚1530.8681640625 proc.memory.rssMB‚ 2841.5625* proc.memory.percent‚17.343521118164062 cpu‚26.078909858928196 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0% disk.nvme1n1.out‚3309.19677734375 disk.md0.in‚0" disk.dm-0.out‚11061.1474609375 disk.rbd0.out‚0% disk.nvme0n1.out‚9348.24951171875# memory_percent‚98.4919786453247X]åžž ‚(:“ è¼ôÒ¾†«Ì gpu.0.gpu‚98.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚266.046' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.68199999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 *$2Ÿ - _wandbruntime‚1006  _runtime‚1006X†²íÈ   ' lr-AdamW/pg2‚0.00021236034141885665 trainer/global_step‚1999 $ _timestamp‚1.784487530086991e+09 _runtime‚1006.661963275 _step‚158 ' lr-AdamW/pg1‚0.00021236034141885665sç~Í¡ Ç $ _timestamp‚1.784487530086991e+09 _runtime‚1006.661963275 _step‚158 ' lr-AdamW/pg1‚0.00021236034141885665 ' lr-AdamW/pg2‚0.00021236034141885665 trainer/global_step‚1999žd¼î(2¢ - _wandbruntime‚1006  _runtime‚1006’V<˜£ ÜO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1910638518784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚152611382512179 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚154208163507200 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1910638518784 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870272 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚152297524988416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10536030 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1970109830144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15490312 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20158288 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122942451123712 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1907195164579 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚152297524988416 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚154207805202944 GADeviceStatsžÕM JMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚122942304014336 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15779620 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚150704170308240 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20148146 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15779903 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35638456 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15490310 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870272 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35648600 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5243590 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 $ _timestamp‚1.784487530087921e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚152297507469824 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1969987145216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122684928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20158288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚120972341293568 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 _step‚159g:Žç'¤ áO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494208 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚152297507469824 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923318 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304256 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚154207805202944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚122942304014336 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15779620 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚150704170308240 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20148146 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15779903 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35638456 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15490310 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870272 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35648600 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5243590 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 $ _timestamp‚1.784487530087921e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚152297507469824 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1969987145216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122684928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20158288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚120972341293568 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 _step‚159 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1910638518784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚152611382512179 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚154208163507200 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1910638518784 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870272 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚152297524988416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10536030 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1970109830144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15490312 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20158288 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122942451123712 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1907195164579 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚152297524988416 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1911227644416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311424 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5246088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚121031217626112 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚152685379855217 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792832 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35655833 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20158021 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚154283285194752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1911227644416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚152371716757504 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10541113 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1911568437248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15497814 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1971068018176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1970953729024 _step‚160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚123002146930688 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 % _timestamp‚1.7844875304448206e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15497812 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 114289152 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870272 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35665991 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚152371716757504 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10541111 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5246364 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15787199 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15497814 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15497812 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20168177 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35665991 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923430 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1907783464497 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚154282926883328 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚123002285644288 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35655833 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20158021 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚121031193201664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1908123387927 _runtime‚1007.019961151 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚152371699238912 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 138713600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚150777596390720 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962790 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001083 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15497812 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20168177 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35665991 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923430 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1907783464497 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚154282926883328 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚123002285644288 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35655833 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20158021 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚121031193201664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1908123387927 _runtime‚1007.019961151 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚152371699238912 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 138713600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚150777596390720 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962790 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001083 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1911227644416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311424 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5246088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚121031217626112 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚152685379855217 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792832 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35655833 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20158021 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚154283285194752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1911227644416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚152371716757504 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10541113 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1911568437248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15497814 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DevL¥pe iceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1971068018176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1970953729024 _step‚160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚123002146930688 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 % _timestamp‚1.7844875304448206e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15497812 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 114289152 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870272 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35665991 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚152371716757504 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10541111 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5246364 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15787199 Í‚¨^2¨ - _wandbruntime‚1007  _runtime‚1007?ÊD© ý /loss/FBt1/detection_loss‚0.6799154511065569 .loss/FBt2/detection_loss‚1.757326262254667 epoch‚2 trainer/global_step‚1999 /loss/FRt2/assignment_loss‚2.918262682933888 1loss/FBt2/assignment_loss‚0.04825348246907067 -loss/FRt1/detection_loss‚1.57604432624213 &loss/total_loss‚12.068755266351328 _runtime‚1007.020572781 _step‚161 1loss/FBt1/assignment_loss‚0.08043249900584301 /loss/FRt2/detection_loss‚0.5794405547548066 1loss/SRqqt1/detection_loss‚0.7125401511917406 % _timestamp‚1.7844875304457874e+09 0loss/FRt1/assignment_loss‚1.1826085608546473 2loss/SRqqt1/assignment_loss‚0.6565968816394057 1loss/SRqqt2/assignment_loss‚0.811957242067309 1loss/SRqqt2/detection_loss‚1.0653771718312626d_œGª ‚ _step‚161 1loss/FBt1/assignment_loss‚0.08043249900584301 /loss/FRt2/detection_loss‚0.5794405547548066 1loss/SRqqt1/detection_loss‚0.7125401511917406 % _timestamp‚1.7844875304457874e+09 0loss/FRt1/assignment_loss‚1.1826085608546473 2loss/SRqqt1/assignment_loss‚0.6565968816394057 1loss/SRqqt2/assignment_loss‚0.811957242067309 1loss/SRqqt2/detection_loss‚1.0653771718312626 /loss/FBt1/detection_loss‚0.6799154511065569 .loss/FBt2/detection_loss‚1.757326262254667 epoch‚2 trainer/global_step‚1999 /loss/FRt2/assignment_loss‚2.918262682933888 1loss/FBt2/assignment_loss‚0.04825348246907067 -loss/FRt1/detection_loss‚1.57604432624213 &loss/total_loss‚12.068755266351328 _runtime‚1007.020572781¡\ÞP« ‚(:Å ÷¼ôÒÂÔëË* proc.memory.percent‚17.343521118164062 cpu‚26.074384023862624 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375' disk.nvme1c1n1.out‚3309.72412109375 disk.nvme1n1.in‚0 network.sent‚ 2.495401e+06 network.recv‚315121 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚11063.71728515625 disk.dm-1.in‚0 disk.rbd0.out‚0% disk.nvme0c0n1.out‚9350.716796875# disk.nvme0n1.out‚9350.716796875) disk./.usagePercent‚76.62854737401533% disk./.usageGB‚1176.4397735595703 disk.md0.in‚0 disk.md0.out‚ 30.890625 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0$ memory_percent‚98.49236011505127 proc.cpu.threads‚17 disk.dm-1.out‚1531.013671875% disk.nvme1n1.out‚3309.72412109375( proc.memory.availableMB‚ 247.01171875 proc.memory.rssMB‚ 2841.5625ªT/Dž¬ ‚(:“ ÷¼ôÒîÀ¸Ì gpu.0.gpu‚91.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚129.017' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚43.00566666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Aù®2­ - _wandbruntime‚1030  _runtime‚1030qÉ® à % _timestamp‚1.7844875537149878e+09 _runtime‚1030.289993268 _step‚162 ' lr-AdamW/pg1‚0.00021231871327769507 ' lr-AdamW/pg2‚0.00021231871327769507 trainer/global_step‚2049–Âyί È % _timestamp‚1.7844875537149878e+09 _runtime‚1030.289993268 _step‚162 ' lr-AdamW/pg1‚0.00021231871327769507 ' lr-AdamW/pg2‚0.00021231871327769507 trainer/global_step‚2049¢¾ÑãH2° - _wandbruntime‚1030  _runtime‚1030¤$Ûã'± ÝO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304256 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15865412 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1957166340608 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1956825554944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚156007095924224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1956825554944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 _step‚163 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870272 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15865410 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16160553 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚157963921479168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156007113442816 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚154375491471600 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16160830 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 118490624 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20643748 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚156329129701187 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2017998073344 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15865410 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚123916157920768 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962678 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚156328772738509 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1953638229587 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚157963921479168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20653890 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 142915072 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚1030.294409975 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494208 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚125934274484736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚123916133496320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36519302 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10790080 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5370473 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2018116563968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923318 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36509158 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚157964279783424 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10790082 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1956825554944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚156007095924224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1956825554944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 _step‚163 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870272 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15865410 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16160553 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚157963921479168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156007113442816 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚154375491471600 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16160830 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 118490624 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20643748 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚156329129701187 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2017998073344 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15865410 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚123916157920768 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962678 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚156328772738509 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1953638229587 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚157963921479168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20653890 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 142915072 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚1030.294409975 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494208 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚125934274484736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚123916133496320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36519302 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10790080 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5370473 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2018116563968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923318 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36509158 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚157964279783424 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10790082 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156081305211904 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10795161 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962790 trainer/global_step‚2049 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚158039401470976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36536693 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149199360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001083 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1957755466240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚123975009828864 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16168126 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5372965 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15872914 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15872912 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20653623 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494208 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1958096259072 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1957755466240 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20653623 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KArÉ×EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚158039043159552 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚156081287693312 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚156081305211904 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792832 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15872914 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 % _timestamp‚1.7844875540741715e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36526535 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923430 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36536693 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792832 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36526535 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚158039401470976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15872912 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2019076695552 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1954226529505 _step‚164 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 % _timestamp‚1.7844875540741715e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36526535 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923430 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36536693 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792832 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36526535 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚158039401470976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15872912 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2019076695552 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1954226529505 _step‚164 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156081305211904 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10795161 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962790 trainer/global_step‚2049 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚158039401470976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36536693 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149199360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001083 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1957755466240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚123975009828864 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16168126 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5372965 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15872914 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15872912 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20653623 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494208 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1958096259072 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1957755466240 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20653623 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚158039043159552 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚156081287693312 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚156081305211904 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792832 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15872914 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264¤V?ÑÝ2¶ - _wandbruntime‚1030  _runtime‚1030/M· ÿ 2loss/SRqqt1/assignment_loss‚0.6908423756650567 1loss/FBt2/assignment_loss‚0.16275567735525934 1loss/SRqqt2/detection_loss‚1.1392010764185605 % _timestamp‚1.7844875540751364e+09 1loss/SRqqt1/detection_loss‚0.7346039555541211 %loss/total_loss‚15.73294637023993 0loss/FRt2/assignment_loss‚2.6285660171854324 /loss/FRt1/detection_loss‚1.3815624568744804 /loss/FBt1/detection_loss‚1.3372333683469546 .loss/FBt2/detection_loss‚5.003195670200512 epoch‚2 _runtime‚1030.649791007 _step‚165 0loss/FRt1/assignment_loss‚1.2224380501200645 2loss/SRqqt2/assignment_loss‚0.7872970099353032 1loss/FBt1/assignment_loss‚0.06286421758146848 .loss/FRt2/detection_loss‚0.582386495002717 trainer/global_step‚2049F÷/Õ ¸ „ /loss/FRt1/detection_loss‚1.3815624568744804 /loss/FBt1/detection_loss‚1.3372333683469546 .loss/FBt2/detection_loss‚5.003195670200512 epoch‚2 _runtime‚1030.649791007 _step‚165 0loss/FRt1/assignment_loss‚1.2224380501200645 2loss/SRqqt2/assignment_loss‚0.7872970099353032 1loss/FBt1/assignment_loss‚0.06286421758146848 .loss/FRt2/detection_loss‚0.582386495002717 trainer/global_step‚2049 2loss/SRqqt1/assignment_loss‚0.6908423756650567 1loss/FBt2/assignment_loss‚0.16275567735525934 1loss/SRqqt2/detection_loss‚1.1392010764185605 % _timestamp‚1.7844875540751364e+09 1loss/SRqqt1/detection_loss‚0.7346039555541211 %loss/total_loss‚15.73294637023993 0loss/FRt2/assignment_loss‚2.6285660171854324¥ÛI¹R¹ ‚(:Ç †½ôÒ’·˜Ì# disk.dm-0.out‚11064.61962890625 disk.dm-1.in‚0 disk.dm-1.out‚1531.091796875 disk.nvme0c0n1.in‚ 4.3359375# disk.nvme0n1.out‚9351.849609375$ memory_percent‚98.49097728729248 proc.memory.rssMB‚ 2841.5625 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚3310.41552734375 disk.md0.out‚ 31.27734375 proc.cpu.threads‚17 network.sent‚ 2.571729e+06) disk./.usagePercent‚76.62859632339564 disk.rbd0.out‚0( proc.memory.availableMB‚ 247.23828125* proc.memory.percent‚17.343521118164062 cpu‚26.069965862735742 network.recv‚319116% disk./.usageGB‚1176.4405250549316% disk.nvme1n1.out‚3310.41552734375 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 disk.rbd0.in‚0% disk.nvme0c0n1.out‚9351.849609375 disk.nvme0n1.in‚ 4.3359375 disk.nvme1n1.in‚0(h机 ‚(:‘ †½ôÒêƒÊÌ gpu.0.gpu‚23.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚263.63' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.87666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£úK^L» ‚(:Á •½ôÒÜÿÇË proc.cpu.threads‚17 network.recv‚323006# disk.nvme0c0n1.out‚ 9353.5234375 disk.md0.out‚ 31.69140625 proc.memory.rssMB‚ 2841.5625* proc.memory.percent‚17.343521118164062 network.sent‚ 2.620131e+06( disk./.usagePercent‚76.6285888691753% disk./.usageGB‚1176.4404106140137 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375' disk.nvme1c1n1.out‚3311.09521484375% disk.nvme1n1.out‚3311.09521484375# disk.dm-0.out‚11065.96630859375 disk.dm-1.in‚0! disk.dm-1.out‚1531.2021484375 disk.rbd0.in‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 247.203125! disk.nvme0n1.out‚ 9353.5234375 disk.nvme1n1.in‚0$ memory_percent‚98.49119186401367 cpu‚26.066191353603894íΔŽž¼ ‚(:“ •½ôÒ†±Ì gpu.0.gpu‚67.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚261.395' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.13166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<¨Hß2½ - _wandbruntime‚1054  _runtime‚1054—1•Ëɾ à ' lr-AdamW/pg1‚0.00021227538902784254 ' lr-AdamW/pg2‚0.00021227538902784254 trainer/global_step‚2099 % _timestamp‚1.7844875776462102e+09 _runtime‚1054.221155311 _step‚166º(%•Î¿ È _step‚166 ' lr-AdamW/pg1‚0.00021227538902784254 ' lr-AdamW/pg2‚0.00021227538902784254 trainer/global_step‚2099 % _timestamp‚1.7844875776462102e+09 _runtime‚1054.221155311¦OEÙÜ2À - _wandbruntime‚1054  _runtime‚1054˜Ý&:ã'Á ÝO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2003694162432 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21139350 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304256 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170688 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚159716684378624 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚159716701897216 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37390004 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 trainer/global_step‚2099 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2003353376768 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16541412 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚161720037755392 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚161720396059648 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚128926102741504 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚159716684378624 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2065995022848 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16240512 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16240510 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 _step‚167 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5497280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7844875776470475e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11044132 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001083 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21149492 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494208 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241371 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚161720396059648 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚158046778556240 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2066128193536 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16541124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11044130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37379860 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚126859950123520 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16240512 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142ÆèÂ×  âO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚159716701897216 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37390004 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 trainer/global_step‚2099 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2003353376768 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16541412 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚161720037755392 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚161720396059648 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚128926102741504 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚159716684378624 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2065995022848 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16240512 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16240510 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 _step‚167 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5497280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7844875776470475e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11044132 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001083 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21149492 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494208 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241371 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚161720396059648 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚158046778556240 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2066128193536 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16541124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11044130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37379860 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚126859950123520 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16240512 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2003694162432 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21139350 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304256 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170688 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚159716684378624 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0§;2@–2à - _wandbruntime‚1054  _runtime‚1054c*Âvì&Ä æM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚159790893666304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37407395 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2004283288064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚128985782539264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚159790876147712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2004283288064 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2004624080896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚159790893666304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚126918826456064 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21159381 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚161795517747200 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚1054.581198137 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚161795159435776 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 trainer/global_step‚2099 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16548901 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5499405 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚126918850880512 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚158120221678080 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37407395 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131066368 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962790 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2001009517943 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311424 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2004624080896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚160120874233233 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 _step‚168 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792832 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16248012 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2067087149568 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16548616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16248014 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5499688 $ _timestamp‚1.784487578006053e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21149225Ws${ñ&Å ëM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚159790893666304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37407395 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2004283288064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚128985782539264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚159790876147712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2004283288064 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2004624080896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚159790893666304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚126918826456064 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21159381 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚161795517747200 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚1054.581198137 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚161795159435776 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 trainer/global_step‚2099 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16548901 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5499405 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚126918850880512 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚158120221678080 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37407395 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131066368 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962790 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2001009517943 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311424 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2004624080896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚160120874233233 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 _step‚168 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792832 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16248012 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2067087149568 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16548616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16248014 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5499688 $ _timestamp‚1.784487578006053e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21149225¨§«ëI2Æ - _wandbruntime‚1054  _runtime‚1054ò*êÇ ý 2loss/SRqqt2/assignment_loss‚0.7567201428553637 /loss/FRt2/detection_loss‚0.5652904501070857 /loss/FBt1/detection_loss‚2.2660956536766093 epoch‚2 _runtime‚1054.581640838 _step‚169 /loss/FRt1/assignment_loss‚1.213486806009755 0loss/FBt2/assignment_loss‚0.2951333083328791 1loss/SRqqt1/detection_loss‚0.8042781563016572 1loss/SRqqt2/detection_loss‚1.0129412369254758 /loss/FBt2/detection_loss‚1.9224220514297485 0loss/FRt2/assignment_loss‚2.8791678007025467 1loss/FBt1/assignment_loss‚0.05703383556536655 $loss/total_loss‚13.8134296284428 2loss/SRqqt1/assignment_loss‚0.5829816831902229 .loss/FRt1/detection_loss‚1.457878503346089 trainer/global_step‚2099 % _timestamp‚1.7844875780070016e+09ïaQÈ ‚ 0loss/FBt2/assignment_loss‚0.2951333083328791 1loss/SRqqt1/detection_loss‚0.8042781563016572 1loss/SRqqt2/detection_loss‚1.0129412369254758 /loss/FBt2/detection_loss‚1.9224220514297485 0loss/FRt2/assignment_loss‚2.8791678007025467 1loss/FBt1/assignment_loss‚0.05703383556536655 $loss/total_loss‚13.8134296284428 2loss/SRqqt1/assignment_loss‚0.5829816831902229 .loss/FRt1/detection_loss‚1.457878503346089 trainer/global_step‚2099 % _timestamp‚1.7844875780070016e+09 2loss/SRqqt2/assignment_loss‚0.7567201428553637 /loss/FRt2/detection_loss‚0.5652904501070857 /loss/FBt1/detection_loss‚2.2660956536766093 epoch‚2 _runtime‚1054.581640838 _step‚169 /loss/FRt1/assignment_loss‚1.213486806009755©T¡–£RÉ ‚(:Ç ¤½ôÒî“ãË disk.md0.out‚ 31.859375 disk.dm-0.in‚ 4.3359375 disk.nvme0c0n1.in‚ 4.3359375& disk.nvme0c0n1.out‚9354.4580078125' disk.nvme1c1n1.out‚3311.46337890625 network.recv‚326267# disk.dm-0.out‚11066.77880859375 disk.dm-1.in‚0! disk.dm-1.out‚1531.3076171875 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0$ memory_percent‚98.49145412445068 proc.memory.rssMB‚ 2841.5625 network.sent‚ 2.638883e+06% disk./.usageGB‚1176.4409408569336 disk.rbd0.in‚0$ disk.nvme0n1.out‚9354.4580078125% disk.nvme1n1.out‚3311.46337890625* proc.memory.percent‚17.343521118164062 cpu‚26.06178461247998) disk./.usagePercent‚76.62862340706292 disk.md0.in‚0 disk.rbd0.out‚0 disk.nvme1n1.in‚0( proc.memory.availableMB‚ 247.16015625 proc.cpu.threads‚17Ô^©žÊ ‚(:“ ¤½ôÒ¿¨šÌ gpu.0.gpu‚70.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚265.902' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.63399999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MÕ;2Ë - _wandbruntime‚1078  _runtime‚1078~a:‰ÉÌ Ã trainer/global_step‚2149 % _timestamp‚1.7844876016767895e+09 _runtime‚1078.251761357 _step‚170 ' lr-AdamW/pg1‚0.00021223036936322476 ' lr-AdamW/pg2‚0.00021223036936322476ÕîìÎÍ È trainer/global_step‚2149 % _timestamp‚1.7844876016767895e+09 _runtime‚1078.251761357 _step‚170 ' lr-AdamW/pg1‚0.00021223036936322476 ' lr-AdamW/pg2‚0.00021223036936322476ª†Ô~ä2Î - _wandbruntime‚1078  _runtime‚1078’íE¦V Ï ÝO G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚165476512335872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2046524359603 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21634952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11298180 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16615610 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚171 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚165476154031616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923318 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21645094 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚163764267116525 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2046184436285 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5624733 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001083 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚165476154031616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16615612 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0Ž€Oè'Ð âO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16615610 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚171 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚165476154031616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923318 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21645094 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚163764267116525 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2046184436285 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5624733 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001083 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚165476154031616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16615612 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚165476512335872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2046524359603 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21634952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11298180 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 I6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2047452582951 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962790 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16930216 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5626955 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38278097 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚163838621422241 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16623114 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21654983 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792832 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚165551275712000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38278097 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16623114 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚161791508762720 _step‚172 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚163838978385031 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚165551634023424 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21644827 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚131977732630528 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2115065122816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚129862667507712 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001083 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚163500482120704 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2050811109888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚131977579236864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2050811109888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 trainer/global_step‚2149 _runtime‚1078.610116374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11303261™MÎñIÓ ìM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128969216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚163500464602112 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚129862643083264 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚165551634023424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5627252 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38267939 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792832 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494208 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16623112 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚299 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚131977579236864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2050811109888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 trainer/global_step‚2149 _runtime‚1078.610116374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11303261 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38267939 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2047452582951 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.U3Åo©small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962790 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16930216 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5626955 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38278097 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚163838621422241 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16623114 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21654983 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792832 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚165551275712000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38278097 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16623114 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚161791508762720 _step‚172 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚163838978385031 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚165551634023424 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21644827 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚131977732630528 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2115065122816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚129862667507712 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001083 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚163500482120704 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2050811109888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚134909719876096 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚167482371268211 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001083 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2096749806080 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303680 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚175 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚300 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2161985194496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962678 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22140696 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22130554 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 _runtime‚ 1102.19638621 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚169232270307840 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11552232 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚132747607802368 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2162112073728 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 trainer/global_step‚2199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2096409020416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚167482014305533 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126879232 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚169232270307840 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870272 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39131408 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22140696 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17303384 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5751452 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2096749806080 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚132747583377920 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16990710 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494208 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚167135861287424 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17303684 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304256 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚298 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2092627501293 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494208 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚134909719876096 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚167482371268211 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001083 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2096749806080 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303680 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚175 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚300 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2161985194496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962678 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22140696 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22130554 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 _runtime‚ 1102.19638621 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚169232270307840 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11552232 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚132747607802368 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2162112073728 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 trainer/global_step‚2199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2096409020416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚167482014305533 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126879232 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚169232270307840 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870272 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39131408 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22140696 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17303384 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5751452 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2096749806080¯‘q22á - _wandbruntime‚1102  _runtime‚1102˜‡[ÑÞâ çM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16998214 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792832 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17311307 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2093895647959 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39138641 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚167210070575104 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚132806459710464 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚167210053056512 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 120580608 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚167210053056512 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2097338931712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001083 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962790 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16998212 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 145005056 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 _runtime‚1102.552730309 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚167210070575104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870272 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39148799 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 trainer/global_step‚2199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚169307750299648 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11557313 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚167556725574039 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5753712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22140429 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17311023 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494208 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22150585 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2162946234368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39148799 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22150585 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2097338931712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134969550949888 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚169307391988224 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22140429 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚169307750299648 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11557311 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134969405944832 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7844876259776263e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592ûüÅò&ã ìM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11557311 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134969405944832 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7844876259776263e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16998214 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792832 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17311307 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2093895647959 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39138641 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚167210070575104 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚132806459710464 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚167210053056512 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 120580608 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚167210053056512 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2097338931712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001083 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962790 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16998212 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 145005056 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 _runtime‚1102.552730309 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚167210070575104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870272 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39148799 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 trainer/global_step‚2199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚169307750299648 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11557313 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚167556725574039 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5753712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22140429 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17311023 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494208 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22150585 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2162946234368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39148799 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22150585 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2097338931712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134969550949888 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚169307391988224 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22140429 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚169307750299648° Úí2ä - _wandbruntime‚1102  _runtime‚1102ŒÕDå ÿ &loss/total_loss‚12.987279116875609 trainer/global_step‚2199 0loss/FBt1/assignment_loss‚0.2561012327844853 1loss/FBt2/assignment_loss‚0.02126180018636926 0loss/SRqqt1/detection_loss‚0.766871255610032 1loss/SRqqt2/detection_loss‚1.2769829016178846 % _timestamp‚1.7844876259785678e+09 _runtime‚1102.553230399 /loss/FRt2/detection_loss‚0.5298929575644334 /loss/FBt2/detection_loss‚1.4513860446485607 epoch‚2 /loss/FRt1/assignment_loss‚1.317604514237108 /loss/FRt1/detection_loss‚1.4441146053068725 _step‚177 0loss/FRt2/assignment_loss‚2.8302096419526768 2loss/SRqqt1/assignment_loss‚0.5304037803687234 2loss/SRqqt2/assignment_loss‚0.6879662824522731 /loss/FBt1/detection_loss‚1.8744841001461898‘ rÄ æ „ 0loss/FBt1/assignment_loss‚0.2561012327844853 1loss/FBt2/assignment_loss‚0.02126180018636926 0loss/SRqqt1/detection_loss‚0.766871255610032 1loss/SRqqt2/detection_loss‚1.2769829016178846 % _timestamp‚1.7844876259785678e+09 _runtime‚1102.553230399 /loss/FRt2/detection_loss‚0.5298929575644334 /loss/FBt2/detection_loss‚1.4513860446485607 epoch‚2 /loss/FRt1/assignment_loss‚1.317604514237108 /loss/FRt1/detection_loss‚1.4441146053068725 _step‚177 0loss/FRt2/assignment_loss‚2.8302096419526768 2loss/SRqqt1/assignment_loss‚0.5304037803687234 2loss/SRqqt2/assignment_loss‚0.6879662824522731 /loss/FBt1/detection_loss‚1.8744841001461898 &loss/total_loss‚12.987279116875609 trainer/global_step‚2199±_œåLBç ‚(:· ѽôÒêþÉË% disk.nvme1n1.out‚3324.22509765625 disk.md0.in‚0% proc.memory.availableMB‚ 246.96875* proc.memory.percent‚17.343521118164062 disk.dm-1.in‚0 disk.nvme1n1.in‚0 disk.dm-0.in‚ 4.3359375' disk.nvme1c1n1.out‚3324.22509765625% disk./.usageGB‚1176.4992790222168" disk.dm-1.out‚1531.82568359375 disk.nvme0n1.out‚ 9380.90625 disk.nvme1c1n1.in‚0" disk.dm-0.out‚11102.2333984375$ memory_percent‚98.49262237548828 proc.cpu.threads‚17 network.sent‚ 2.754442e+06 network.recv‚337947! disk.nvme0c0n1.out‚ 9380.90625 disk.nvme0n1.in‚ 4.3359375 disk.md0.out‚33.375 proc.memory.rssMB‚ 2841.5625 cpu‚26.04997792297428) disk./.usagePercent‚76.63242332012486 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.3359375EY±õœè ‚(:‘ ѽôÒ®Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚118.901' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.63366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦ð~RSé ‚(:È à½ôÒÞ³ÉË$ disk.nvme1n1.out‚3324.6806640625 disk.md0.in‚0 disk.md0.out‚ 33.6328125' proc.memory.availableMB‚ 246.9453125 proc.memory.rssMB‚ 2841.5625* proc.memory.percent‚17.343521118164062 cpu‚26.046246827576695 network.recv‚341960& disk.nvme1c1n1.out‚3324.6806640625 proc.cpu.threads‚17 network.sent‚ 2.818056e+06% disk./.usageGB‚1176.4992446899414" disk.dm-0.out‚11138.0732421875 disk.rbd0.out‚0 disk.nvme0n1.in‚ 4.3359375% disk.nvme0n1.out‚9416.96337890625$ memory_percent‚98.49276542663574) disk./.usagePercent‚76.63242108385874 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0" disk.dm-1.out‚1531.92333984375 disk.nvme0c0n1.in‚ 4.3359375' disk.nvme0c0n1.out‚9416.96337890625 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.rbd0.in‚0Â!‹‘ê ‚(:† à½ôÒ¿¥‹Ì gpu.0.gpu‚70.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚254.13' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚84.71 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Vè%É2ë - _wandbruntime‚1126  _runtime‚1126>‹DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001083 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40002110 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2142936842240 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870272 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22626156 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11806282 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚179 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304256 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5878085 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2139410489619 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17365812 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22636298 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22626156 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170845467260416 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚137901539260416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157595136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172988386584064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304256 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚169060690928240 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚172988386584064 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚172988744888320 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2143277627904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2209981660160 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚169060707967600 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962678 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚137901381665280 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚2249 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870272 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22626156 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11806282 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚179 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304256 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5878085 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2139410489619 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17365812 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22636298 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22626156 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170845467260416 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚137901539260416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157595136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172988386584064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304256 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚169060690928240 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚172988386584064 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚172988744888320 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2143277627904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2209981660160 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚169060707967600 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962678 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚137901381665280 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚2249 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001083 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934³R¼52ñ - _wandbruntime‚1126  _runtime‚1126!ïwkë&ò åM ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚169134117010720 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40019501 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40009343 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22636031 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17373314 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170919659029504 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2143866753536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 _step‚180 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚170919641510912 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792832 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128969216 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚169134134050080 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11811361 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962790 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135750276337664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22646187 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22636031 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17373312 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2211067917312 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2144207546368 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17691975 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚171274472763047 % _timestamp‚1.7844876499234192e+09 _runtime‚ 1126.4986077 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494208 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17373314 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 trainer/global_step‚2249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17692248 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5880614 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40019501 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2210938948096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚173063866575872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135750300762112 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚173063866575872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2144207546368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚137961215285760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001083 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22646187 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚171274115800257 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2139998789537 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17373312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170919659029504 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5880885 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11811363 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚137961368679424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40009343 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675´)…Vð&ó êM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11811361 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962790 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135750276337664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22646187 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22636031 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17373312 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2211067917312 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2144207546368 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17691975 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚171274472763047 % _timestamp‚1.7844876499234192e+09 _runtime‚ 1126.4986077 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494208 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17373314 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 trainer/global_step‚2249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17692248 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5880614 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40019501 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2210938948096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚173063866575872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135750300762112 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚173063866575872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2144207546368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚137961215285760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001083 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792832 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22646187 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚171274115800257 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2139998789537 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17373312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170919659029504 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5880885 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11811363 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚137961368679424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40009343 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚169134117010720 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40019501 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40009343 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22636031 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17373314 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170919659029504 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2143866753536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 _step‚180 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚170919641510912 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792832 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128969216 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚169134134050080´Θê2ô - _wandbruntime‚1126  _runtime‚1126h“)îõ € 2loss/SRqqt1/assignment_loss‚0.6010040073461981 1loss/FBt1/assignment_loss‚0.07961021066502853 2loss/FBt2/assignment_loss‚0.011175259486459291 /loss/FRt1/detection_loss‚1.5208243743561387 /loss/FRt2/detection_loss‚0.6140064143266829 trainer/global_step‚2249 0loss/FRt1/assignment_loss‚1.0226832628250122 /loss/FBt1/detection_loss‚0.6637019493178747 /loss/FBt2/detection_loss‚2.3596043870784342 epoch‚2 .loss/FRt2/assignment_loss‚2.74696583529445 1loss/SRqqt1/detection_loss‚0.6674437910119525 &loss/total_loss‚12.386679833528172 % _timestamp‚1.7844876499243977e+09 _step‚181 2loss/SRqqt2/assignment_loss‚0.8577611795251511 0loss/SRqqt2/detection_loss‚1.241899162294789 _runtime‚1126.499094181G«ú ö … epoch‚2 .loss/FRt2/assignment_loss‚2.74696583529445 1loss/SRqqt1/detection_loss‚0.6674437910119525 &loss/total_loss‚12.386679833528172 % _timestamp‚1.7844876499243977e+09 _step‚181 2loss/SRqqt2/assignment_loss‚0.8577611795251511 0loss/SRqqt2/detection_loss‚1.241899162294789 _runtime‚1126.499094181 2loss/SRqqt1/assignment_loss‚0.6010040073461981 1loss/FBt1/assignment_loss‚0.07961021066502853 2loss/FBt2/assignment_loss‚0.011175259486459291 /loss/FRt1/detection_loss‚1.5208243743561387 /loss/FRt2/detection_loss‚0.6140064143266829 trainer/global_step‚2249 0loss/FRt1/assignment_loss‚1.0226832628250122 /loss/FBt1/detection_loss‚0.6637019493178747 /loss/FBt2/detection_loss‚2.3596043870784342µùåCäO÷ ‚(:Ä ï½ôÒÒ—ÂË disk.dm-1.out‚1532.177734375% disk.nvme0n1.out‚9419.05419921875' disk.nvme1c1n1.out‚3325.56494140625* proc.memory.percent‚17.343521118164062 cpu‚26.0428318783967) disk./.usagePercent‚76.63234952334335$ disk./.usageGB‚1176.498146057129' disk.nvme0c0n1.out‚9419.05419921875 disk.nvme1n1.in‚0& proc.memory.availableMB‚ 246.890625 disk.nvme0c0n1.in‚ 4.3359375 network.sent‚ 2.82205e+06 network.recv‚345157 disk.md0.out‚ 34.05078125 disk.dm-0.in‚ 4.3359375" disk.dm-0.out‚11139.9169921875 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.md0.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚3325.56494140625$ memory_percent‚98.49309921264648 proc.memory.rssMB‚ 2841.5625 proc.cpu.threads‚17Ô4H$óø ‚(:“ ï½ôÒ˜”¢Ì gpu.0.gpu‚70.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚271.103' gpu.0.enforcedPowerLimitWatts‚½™l÷«300.0( gpu.0.powerPercent‚90.36766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼£Ø2ù - _wandbruntime‚1149  _runtime‚1149ÌbÈú  ' lr-AdamW/pg2‚0.00021208514522730115 trainer/global_step‚2299 $ _timestamp‚1.784487673224417e+09 _runtime‚1149.799386404 _step‚182 ' lr-AdamW/pg1‚0.00021208514522730115àŸÞ¢Íû Ç ' lr-AdamW/pg1‚0.00021208514522730115 ' lr-AdamW/pg2‚0.00021208514522730115 trainer/global_step‚2299 $ _timestamp‚1.784487673224417e+09 _runtime‚1149.799386404 _step‚182¶ :·2ü - _wandbruntime‚1149  _runtime‚1149pu¶ùâ'ý ÜO _step‚183 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚174555055714816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176744502860288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40872812 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚174917508683549 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12060330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18064475 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚174555038196224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2189464664064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23131900 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚176744861164544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304256 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785664 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241371 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 _runtime‚1149.800191475 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2189464664064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚176744502860288 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2185513631309 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6004145 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23121758 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962678 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785664 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494208 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40862668 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚140893187272192 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17740912 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304256 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23121758 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6004428 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001083 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚174917865646227 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2189805449728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2185853554627 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17740912 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2258097519104 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2257970639872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174555055714816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚172731995052240 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18064760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126879232 trainer/global_step‚2299 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚138635216632320 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17740910 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923318 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176744502860288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40872812 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚174917508683549 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12060330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18064475 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚174555038196224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2189464664064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23131900 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚176744861164544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304256 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785664 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241371 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 _runtime‚1149.800191475 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2189464664064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚176744502860288 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2185513631309 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6004145 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23121758 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962678 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785664 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494208 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40862668 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚140893187272192 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17740912·ÛªJ2ÿ - _wandbruntime‚1150  _runtime‚1150]Ðkíì&€ æM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792832 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚172805421134720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚176819982852096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2190394575360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23131633 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140953174065152 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17748414 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 147102208 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2186781777975 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12065413 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚176819982852096 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6006706 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚174629229965312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 1150.15883599 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001083 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17748414 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2190735368192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12065411 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2258933998080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 trainer/global_step‚2299 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2190735368192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962790 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17748412 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494208 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40880045 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2259056675840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122677760 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚172805438174080 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40880045 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚174629247483904 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚176819624540672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7844876735837252e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23131633 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚174629247483904 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40890203 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17748412 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40890203 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚174992219952055 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚174991862989265 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2190394575360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚138694117389312 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18072406 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18072117 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23141789 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140953026962944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923430 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259¿x”b/ ëM YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚176819624540672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7844876735837252e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23131633 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚174629247483904 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40890203 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17748412 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40890203 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDevicàóó eStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚174992219952055 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚174991862989265 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2190394575360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚138694117389312 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18072406 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18072117 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23141789 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140953026962944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923430 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792832 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚172805421134720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚176819982852096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2190394575360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23131633 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140953174065152 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17748414 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 147102208 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2186781777975 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12065413 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚176819982852096 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6006706 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚174629229965312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 1150.15883599 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001083 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17748414 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2190735368192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12065411 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2258933998080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 trainer/global_step‚2299 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2190735368192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962790 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17748412 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494208 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40880045 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2259056675840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122677760 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚172805438174080 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40880045 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚174629247483904 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272¸ÌrÌ2‚ - _wandbruntime‚1150  _runtime‚1150ÆU;ƒ ÿ 0loss/FRt1/assignment_loss‚1.3200877607914439 1loss/FBt1/assignment_loss‚0.03356941733368964 0loss/SRqqt1/detection_loss‚0.735144240369142 /loss/FBt1/detection_loss‚0.8541279853008711 &loss/total_loss‚12.565166083157864 $ _timestamp‚1.784487673584671e+09 /loss/FRt1/detection_loss‚1.4171798830749793 /loss/FRt2/detection_loss‚0.5800470248278198 epoch‚2 _runtime‚ 1150.15932153 _step‚185 2loss/SRqqt2/assignment_loss‚0.8390516738330422 1loss/FBt2/assignment_loss‚0.10400785728477656 /loss/FBt2/detection_loss‚2.1263119212484787 0loss/FRt2/assignment_loss‚2.8479776818176794 2loss/SRqqt1/assignment_loss‚0.6481075469079685 1loss/SRqqt2/detection_loss‚1.0595530903679722 trainer/global_step‚2299'Œqî „ „ _runtime‚ 1150.15932153 _step‚185 2loss/SRqqt2/assignment_loss‚0.8390516738330422 1loss/FBt2/assignment_loss‚0.10400785728477656 /loss/FBt2/detection_loss‚2.1263119212484787 0loss/FRt2/assignment_loss‚2.8479776818176794 2loss/SRqqt1/assignment_loss‚0.6481075469079685 1loss/SRqqt2/detection_loss‚1.0595530903679722 trainer/global_step‚2299 0loss/FRt1/assignment_loss‚1.3200877607914439 1loss/FBt1/assignment_loss‚0.03356941733368964 0loss/SRqqt1/detection_loss‚0.735144240369142 /loss/FBt1/detection_loss‚0.8541279853008711 &loss/total_loss‚12.565166083157864 $ _timestamp‚1.784487673584671e+09 /loss/FRt1/detection_loss‚1.4171798830749793 /loss/FRt2/detection_loss‚0.5800470248278198 epoch‚2¹uÄuU… ‚(:Ê þ½ôÒÝê‚Ì' disk.nvme1c1n1.out‚3326.09228515625 disk.md0.in‚0 disk.dm-1.out‚1532.279296875' disk.nvme0c0n1.out‚9419.89404296875% disk.nvme0n1.out‚9419.89404296875 disk.nvme1n1.in‚0 disk.md0.out‚ 34.33984375$ memory_percent‚98.49350452423096 cpu‚26.039050433141178 proc.cpu.threads‚17) disk./.usagePercent‚76.63234256607103 disk.dm-0.in‚ 4.3359375 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375( proc.memory.availableMB‚ 246.82421875* proc.memory.percent‚17.343521118164062 network.sent‚ 2.898311e+06% disk./.usageGB‚1176.4980392456055" disk.dm-0.out‚11140.5498046875 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚3326.09228515625 proc.memory.rssMB‚ 2841.5625 network.recv‚349660 disk.rbd0.in‚0 disk.rbd0.out‚03s‰5ž† ‚(:“ þ½ôÒÌ͹Ì gpu.0.gpu‚91.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚261.181' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.06033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š6.ÕO‡ ‚(:Ä ¾ôÒ¯ûíË disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1n1.in‚0" disk.dm-1.out‚1532.41259765625 proc.memory.rssMB‚ 2841.5625 cpu‚26.036029193516917 disk.md0.in‚0* proc.memory.percent‚17.343521118164062 network.sent‚ 2.948586e+06% disk.nvme0c0n1.out‚9422.244140625 disk.nvme0n1.in‚ 4.3359375# disk.nvme0n1.out‚9422.244140625 disk.nvme1c1n1.in‚0$ memory_percent‚98.49352836608887' proc.memory.availableMB‚ 246.8203125 proc.cpu.threads‚17) disk./.usagePercent‚76.63241015100223% disk./.usageGB‚1176.4990768432617' disk.nvme1c1n1.out‚3327.02392578125 disk.dm-0.in‚ 4.3359375 disk.dm-0.out‚ 11142.7109375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme1n1.out‚3327.02392578125 disk.md0.out‚ 34.80078125 network.recv‚353220ZF¤»žˆ ‚(:“ ¾ôÒúÃ¥Ì gpu.0.gpu‚91.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚54.0 gpu.0.powerWatts‚147.457' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚49.15233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)à2‰ - _wandbruntime‚1173  _runtime‚1173r”CœÉŠ à _step‚186 ' lr-AdamW/pg1‚0.00021203335138581485 ' lr-AdamW/pg2‚0.00021203335138581485 trainer/global_step‚2349 % _timestamp‚1.7844876971952772e+09 _runtime‚1173.770283956͵Ï΋ È ' lr-AdamW/pg1‚0.00021203335138581485 ' lr-AdamW/pg2‚0.00021203335138581485 trainer/global_step‚2349 % _timestamp‚1.7844876971952772e+09 _runtime‚1173.770283956 _step‚186ºµŽKÛ2Œ - _wandbruntime‚1173  _runtime‚1173øªâ' ÜO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚178264626650624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141579033259520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41743514 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12314380 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚180500619136512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚178635255872557 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2231956696317 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 _step‚187 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚180500977440768 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785664 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23627502 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 trainer/global_step‚2349 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12314382 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241371 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001083 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23617360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2306108239872 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚176403299176240 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2236333271552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18446137 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 _runtime‚1173.771185477 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2232296619635 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚180500977440768 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2235992485888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6131463 $ _timestamp‚1.784487697196288e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2236333271552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141579057683968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18445843 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚180500619136512 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18116012 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073536 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41733370 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2235992485888 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚178264644169216 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚178635612835235 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18116010 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚178264626650624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923318 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23627502 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18116010 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143885010425856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368ÈÕi}Þ$Ž áO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚178264644169216 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚178635612835235 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18116010 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚178264626650624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923318 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23627502 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18116010 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143885010425856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚178264626650624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141579033259520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41743514 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12314380 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚180500619136512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚178635255872557 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2231956696317 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 _step‚187 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚180500977440768 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340785664 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23627502 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 trainer/global_step‚2349 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12314382 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241371 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001083 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23617360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2306108239872 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚176403299176240 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2236333271552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18446137 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 _runtime‚1173.771185477 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2232296619635 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚180500977440768 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2235992485888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6131463 $ _timestamp‚1.784487697196288e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2236333271552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141579057683968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18445843 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚180500619136512 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18116012 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073536 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41733370 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2235992485888 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302»ñÊòî2 - _wandbruntime‚1174  _runtime‚1174^¯Î'ì& æM $ _timestamp‚1.784487697555731e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41760905 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18123512 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962790 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23637391 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12319463 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41760905 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18123514 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2233224842983 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18453777 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚178338818419712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23637391 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12319461 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2237263190016 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2236922397184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6134018 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2306930542592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18453479 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚180576099128320 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚178338835938304 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚178709967141063 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311424 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚178338835938304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143944840134656 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141637934016512 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934Mg^[ñ&‘ ëM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18123514 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18453777 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚178338818419712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23637391 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12319461 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2237263190016 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2236922397184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6134018 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2306930542592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18453479 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚180576099128320 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚178338835938304 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚178709967141063 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311424 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚178338835938304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143944840134656 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141637934016512 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 $ _timestamp‚1.784487697555731e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41760905 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18123512 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962790 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23637391 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12319463 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41760905 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18123514 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2233224842983 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18491112 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42604072 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24123104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚184256735412736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6258823 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18491112 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870272 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18827253 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24112962 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚182353003061565 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2354101214208 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚182353360024243 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494208 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚144522849886720 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 _runtime‚ 1197.69209308 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24112962 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18491110 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12568432 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923318 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001083 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚144522874311168 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚181974215105024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24123104 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2353976432128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚146876826318848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2282861093376 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6259114 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2278739684643 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962678 trainer/global_step‚2399 _step‚191 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18491110 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18827546 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124782080 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 ^JDeviceStatsMonitor.on_train_batch_start/inS§P@tactive_split_bytes.all.allocated‚146876975525376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚184256735412736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚181974215105024 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚184257093716992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2282520307712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2282520307712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚181974232623616 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42614216 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚180074620339600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚184257093716992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0A•l ç'œ áO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870272 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18827253 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24112962 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚182353003061565 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2354101214208 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚182353360024243 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494208 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚144522849886720 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 _runtime‚ 1197.69209308 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24112962 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18491110 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12568432 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923318 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001083 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚144522874311168 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚181974215105024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24123104 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2353976432128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚146876826318848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2282861093376 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6259114 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2278739684643 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962678 trainer/global_step‚2399 _step‚191 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18491110 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18827546 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124782080 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚146876975525376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚184256735412736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚181974215105024 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚184257093716992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2282520307712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2282520307712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚181974232623616 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42614216 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚180074620339600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚184257093716992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340785664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304256 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18491112 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42604072 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24123104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚184256735412736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6258823 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18491112¿@$2 - _wandbruntime‚1198  _runtime‚1198Fº™/í&ž çM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24122837 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚180148029382720 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42621449 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚299 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2283450219008 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24132993 % _timestamp‚1.7844877214760268e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18834843 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42631607 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6261629 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146936811768320 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚144581726219264 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2279667907991 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42631607 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18498614 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2283791011840 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792832 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962790 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚182427357367281 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚184332215404544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚182427714330071 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001083 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923430 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚184332215404544 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2355061124608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128969216 _step‚192 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18498612 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12573511 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚182048424392704 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚182048424392704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18498614 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚297 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚180148046422080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792832 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24122837 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚2399 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 _runtime‚1198.051143644 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6261332 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24132993 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18835142 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚182048406874112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146936658374656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚182048406874112 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18498612ŽéT¿ò&Ÿ ìM PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18834843 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42631607 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6261629 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146936811768320 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚144581726219264 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2279667907991 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42631607 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18498614 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2283791011840 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340792832 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962790 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚182427357367281 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚184332215404544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚182427714330071 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001083 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923430 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚184332215404544 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2355061124608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128969216 _step‚192 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18498612 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12573511 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241371 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚182048424392704 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚182048424392704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18498614 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚297 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚180148046422080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311424 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340792832 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24122837 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚2399 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 _runtime‚1198.051143644 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870272 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6261332 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24132993 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18835142 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚182048406874112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146936658374656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚182048406874112 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18498612 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24122837 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚180148029382720 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42621449 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚299 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494208 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2283450219008 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24132993 % _timestamp‚1.7844877214760268e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368ÀQˆ½(2  - _wandbruntime‚1198  _runtime‚11985 ¡ û .loss/FBt2/detection_loss‚2.369254932953761 epoch‚2 1loss/SRqqt1/detection_loss‚0.6720993477972476 -loss/FBt1/detection_loss‚0.58696235193013 trainer/global_step‚2399 $ _timestamp‚1.784487721477048e+09 0loss/FRt1/assignment_loss‚1.0576850654054504 2loss/SRqqt2/assignment_loss‚0.7007882711636013 /loss/FRt1/detection_loss‚1.4825786321464671 _runtime‚1198.051691305 2loss/SRqqt1/assignment_loss‚0.6679876474438878 0loss/FBt1/assignment_loss‚0.2061160352093091 2loss/FBt2/assignment_loss‚0.009229659013726632 %loss/total_loss‚11.91436826814397 _step‚193 /loss/FRt2/assignment_loss‚2.739675498438995 .loss/FRt2/detection_loss‚0.559736206924524 1loss/SRqqt2/detection_loss‚0.8622546197168706ÕàóT)¢ € /loss/FRt2/assignment_loss‚2.7O´sÇÝ39675498438995 .loss/FRt2/detection_loss‚0.559736206924524 1loss/SRqqt2/detection_loss‚0.8622546197168706 .loss/FBt2/detection_loss‚2.369254932953761 epoch‚2 1loss/SRqqt1/detection_loss‚0.6720993477972476 -loss/FBt1/detection_loss‚0.58696235193013 trainer/global_step‚2399 $ _timestamp‚1.784487721477048e+09 0loss/FRt1/assignment_loss‚1.0576850654054504 2loss/SRqqt2/assignment_loss‚0.7007882711636013 /loss/FRt1/detection_loss‚1.4825786321464671 _runtime‚1198.051691305 2loss/SRqqt1/assignment_loss‚0.6679876474438878 0loss/FBt1/assignment_loss‚0.2061160352093091 2loss/FBt2/assignment_loss‚0.009229659013726632 %loss/total_loss‚11.91436826814397 _step‚193Á«€=Q£ ‚(:Æ «¾ôÒ©ÿûË disk.nvme1n1.in‚0 disk.dm-1.out‚1532.642578125( proc.memory.availableMB‚ 246.69140625% disk./.usageGB‚1176.1577453613281' disk.nvme1c1n1.out‚3334.06298828125% disk.nvme1n1.out‚3334.06298828125 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0 disk.rbd0.in‚0) disk./.usagePercent‚76.61017719337751 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 disk.dm-0.out‚11153.70703125 disk.rbd0.out‚0# memory_percent‚98.4943151473999 proc.memory.rssMB‚ 2841.5625* proc.memory.percent‚17.343521118164062 network.recv‚359868% disk.nvme0n1.out‚9427.68994140625 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 35.37109375 cpu‚26.03060311898079 proc.cpu.threads‚17 network.sent‚ 3.025916e+06' disk.nvme0c0n1.out‚9427.68994140625ÔàÎYž¤ ‚(:“ «¾ôÒ……³Ì gpu.0.gpu‚65.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚54.0 gpu.0.powerWatts‚217.225' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚72.40833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èì©ÏU¥ ‚(:Ê º¾ôÒ„œçË% disk./.usageGB‚1175.9205856323242 disk.nvme1n1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375$ memory_percent‚98.49896430969238 network.sent‚ 3.077322e+06 network.recv‚364200% disk.nvme1n1.out‚3334.65869140625 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 cpu‚26.062582495984994 proc.cpu.threads‚17 disk.md0.out‚ 35.73046875 disk.dm-1.out‚1532.771484375' disk.nvme0c0n1.out‚9428.93798828125 disk.nvme1c1n1.in‚0' proc.memory.availableMB‚ 245.9296875) disk./.usagePercent‚76.59472956406607 disk.dm-0.out‚11154.64453125 disk.rbd0.out‚0% disk.nvme0n1.out‚9428.93798828125' disk.nvme1c1n1.out‚3334.65869140625# proc.memory.rssMB‚ 2842.22265625) proc.memory.percent‚17.34755039215088ä&I¿¦ ‚(:’ º¾ôÒ¡ñœÌ gpu.0.gpu‚40.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚252.256' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.08533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}´â2§ - _wandbruntime‚1217  _runtime‚1217r[×Ê>¨ •| ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.11541561037302017 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8076617121696472 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.10578305274248123 particle/accuracy_1_of_2‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9892028570175171 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13124999403953552 /particle/accuracy_3_of_4‚0.9914021492004395 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1764288693666458 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.07050960510969162 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9850557446479797 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5404646992683411 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9941036701202393 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.06639498472213745 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4708442687988281 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6077609658241272 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.11518515646457672 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9958896636962891 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5628454089164734 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007046850863844156 particle/accuracy_2_of_2‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9876542687416077 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.08988272398710251 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.599653422832489 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7699999809265137 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9125000238418579 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.46915245056152344 particle/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9643434882164001 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5181020498275757 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17548522353172302 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4745721220970154 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.08738330006599426 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5051934123039246 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8"Purity/*FRt*SRqqt0FBt/event_purity‚0.154326394200325 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4764763414859772 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.06701630353927612 (particle/accuracy‚0.8371155858039856 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 trainer/global_step‚2402 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5785302519798279 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2467755526304245 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.8847925066947937 particle/accuracy_1_of_1‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.10394492745399475 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5432871580123901 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1347333788871765 $particle/f_score‚0.6085165143013 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5189822912216187 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5446044206619263 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.15027514100074768 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2716772258281708 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1445753276348114 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7108331918716431 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8017093539237976 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.556053638458252 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9884464740753174 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8200212121009827 +particle/specificity‚0.9048018455505371 _step‚194 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8923436999320984 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3333333432674408 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.08472882211208344 8"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.344696968793869 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45923101902008057 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5932652354240417 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013000000268220901 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9772727489471436 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5351036787033081 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 *loss/val_total_loss‚11.309463122943125 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.27794355154037476 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1347333788871765 particle/accuracy_2_of_4‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5617262125015259 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5048527121543884 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2986111044883728 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 _runtime‚1217.849452026 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.026271041482686996 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16397753357887268 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.2142857164144516 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7108331918716431 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5007883310317993 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4890700876712799 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5049999952316284 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4672268331050873 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 5 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.96574467420578 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.06874354183673859 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.015929341316223145 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9945660829544067 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.12667998671531677 particle/accuracy_2_of_3‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.17499999701976776 jet/accuracy_5_of_5‚0 % _timestamp‚1.7844877412740073e+09 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.44367071986198425 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.19520525634288788 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.408494234085083 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9941036701202393 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7907894253730774 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_5‚1 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.878433108329773 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5771173238754272 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7699999809265137 +jet/accuracy_1_of_1‚0.21672499179840088 +jet/accuracy_3_of_4‚0.10114290565252304 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5494480729103088 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6175051927566528 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24236395955085754 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9857734441757202 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6486853957176208 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9709851145744324 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9843202233314514 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8499086499214172 *jet/accuracy_1_of_2‚0.6988720297813416 jet/accuracy_4_of_5‚0 particle/accuracy_5_of_5‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_4_of_4‚0.01230158656835556 5 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11184062063694 .particle/accuracy_3_of_3‚0.999514102935791 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1929570883512497 6 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.600694477558136 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5712878704071045 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5600888133049011 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8728623986244202 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7294337749481201 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9562143683433533 /particle/accuracy_4_of_4‚0.9428763389587402 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8928571343421936 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.22963137924671173 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.06731587648391724 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7337719202041626 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9876542687416077 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5888230204582214 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6726495623588562 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9885610938072205 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5165809392929077 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9588843584060669 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.08416356146335602 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.2142857164144516 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4672268331050873 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.2298528105020523 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5321002006530762 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.981812059879303 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015023992396891117 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9953233599662781 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.23600882291793823 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5005525946617126 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.24085232615470886 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.06874354183673859 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2475827932357788 *jet/accuracy_2_of_4‚0.7328525185585022 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13087208569049835 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6526101231575012 *jet/accuracy_2_of_3‚0.4555511474609375 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 jet/accuracy_3_of_5‚1 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.928686797618866 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5481770634651184 6validation_average_jet_accuracy‚0.2744912803173065 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.17499999701976776 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02768934890627861 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1994663029909134 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +particle/sensitivity‚0.5895943641662598 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.49733424186706543 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8023809194564819 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9772727489471436 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.22907450795173645 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22961285710334778 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5128615498542786 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05352955311536789 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.621989905834198 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1154094859957695 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_1_of_4‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5114269852638245 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9921625852584839 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5804777145385742 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 epoch‚2 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9932765960693359 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645·ù3Ö½4© š| +jet/accuracy_2_of_2‚0.11518515646457672 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9958896636962891 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5628454089164734 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007046850863844156 particle/accuracy_2_of_2‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9876542687416077 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.08988272398710251 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.599653422832489 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7699999809265137 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9125000238418579 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.46915245056152344 particle/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9643434882164001 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5181020498275757 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17548522353172302 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4745721220970154 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.08738330006599426 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5051934123039246 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8"Purity/*FRt*SRqqt0FBt/event_purity‚0.154326394200325 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4764763414859772 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.06701630353927612 (particle/accuracy‚0.8371155858039856 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 trainer/global_step‚2402 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5785302519798279 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2467755526304245 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.8847925066947937 particle/accuracy_1_of_1‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.10394492745399475 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5432871580123901 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1347333788871765 $particle/f_score‚0.6085165143013 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5189822912216187 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5446044206619263 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.15027514100074768 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2716772258281708 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1445753276348114 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7108331918716431 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8017093539237976 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.556053638458252 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9884464740753174 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8200212121009827 +particle/specificity‚0.9048018455505371 _step‚194 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8923436999320984 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3333333432674408 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.08472882211208344 8"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.344696968793869 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45923101902008057 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5932652354240417 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013000000268220901 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9772727489471436 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5351036787033081 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 *loss/val_total_loss‚11.309463122943125 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.27794355154037476 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1347333788871765 particle/accuracy_2_of_4‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5617262125015259 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5048527121543884 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2986111044883728 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 _runtime‚1217.849452026 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.026271041482686996 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16397753357887268 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.2142857164144516 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7108331918716431 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5007883310317993 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4890700876712799 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5049999952316284 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4672268331050873 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 5 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.96574467420578 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.06874354183673859 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.015929341316223145 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9945660829544067 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.12667998671531677 particle/accuracy_2_of_3‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.17499999701976776 jet/accuracy_5_of_5‚0 % _timestamp‚1.7844877412740073e+09 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.44367071986198425 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.19520525634288788 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.408494234085083 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9941036701202393 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7907894253730774 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_5‚1 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.878433108329773 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5771173238754272 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7699999809265137 +jet/accuracy_1_of_1‚0.21672499179840088 +jet/accuracy_3_of_4‚0.10114290565252304 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5494480729103088 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6175051927566528 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24236395955085754 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9857734441757202 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6486853957176208 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9709851145744324 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9843202233314514 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8499086499214172 *jet/accuracy_1_of_2‚0.6988720297813416 jet/accuracy_4_of_5‚0 particle/accuracy_5_of_5‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_4_of_4‚0.01230158656835556 5 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11184062063694 .particle/accuracy_3_of_3‚0.999514102935791 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1929570883512497 6 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.600694477558136 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5712878704071045 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5600888133049011 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8728623986244202 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7294337749481201 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9562143683433533 /particle/accuracy_4_of_4‚0.9428763389587402 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8928571343421936 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.22963137924671173 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.06731587648391724 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7337719202041626 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9876542687416077 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5888230204582214 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6726495623588562 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9885610938072205 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5165809392929077 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9588843584060669 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.08416356146335602 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.2142857164144516 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4672268331050873 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.2298528105020523 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5321002006530762 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.981812059879303 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015023992396891117 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9953233599662781 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.23600882291793823 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5005525946617126 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.24085232615470886 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.06874354183673859 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2475827932357788 *jet/accuracy_2_of_4‚0.7328525185585022 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13087208569049835 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6526101231575012 *jet/accuracy_2_of_3‚0.4555511474609375 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 jet/accuracy_3_of_5‚1 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.928686797618866 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5481770634651184 6validation_average_jet_accuracy‚0.2744912803173065 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.17499999701976776 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02768934890627861 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1994663029909134 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +particle/sensitivity‚0.5895943641662598 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.49733424186706543 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8023809194564819 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9772727489471436 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.22907450795173645 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22961285710334778 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5128615498542786 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.0 c 5352955311536789 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.621989905834198 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1154094859957695 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_1_of_4‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5114269852638245 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9921625852584839 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5804777145385742 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 epoch‚2 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9932765960693359 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.11541561037302017 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8076617121696472 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.10578305274248123 particle/accuracy_1_of_2‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9892028570175171 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13124999403953552 /particle/accuracy_3_of_4‚0.9914021492004395 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1764288693666458 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.07050960510969162 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9850557446479797 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5404646992683411 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9941036701202393 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.06639498472213745 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4708442687988281 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6077609658241272 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1Â|ËòrRª ‚(:Ç ɾôÒíÿÊÌ' proc.memory.availableMB‚ 493.6796875 network.sent‚ 3.080244e+06) disk./.usagePercent‚76.59468657806204" disk.dm-1.out‚1778.22998046875 disk.rbd0.out‚0& disk.nvme0c0n1.out‚9443.7177734375$ disk.nvme0n1.out‚9443.7177734375" disk.nvme1n1.out‚ 3581.51171875 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚11170.14697265625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0# proc.memory.rssMB‚ 2845.58984375* proc.memory.percent‚17.368102073669434 cpu‚26.06199463603721 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375$ disk.nvme1c1n1.out‚ 3581.51171875 disk.md0.out‚ 36.03515625 proc.cpu.threads‚17 network.recv‚367503% disk./.usageGB‚1175.9199256896973 disk.nvme1n1.in‚0 disk.md0.in‚0$ memory_percent‚96.98681831359863¤<%¿‘« ‚(:† ɾôÒÊ…ÞË gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚52.0 gpu.0.powerWatts‚118.155' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.385 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—åˆ~2¬ - _wandbruntime‚1241  _runtime‚1241¶$u$É­ à _step‚195 ' lr-AdamW/pg1‚0.00021192468994539593 ' lr-AdamW/pg2‚0.00021192468994539593 trainer/global_step‚2449 % _timestamp‚1.7844877648856406e+09 _runtime‚1241.460658673¬w֮ή È _step‚195 ' lr-AdamW/pg1‚0.00021192468994539593 ' lr-AdamW/pg2‚0.00021192468994539593 trainer/global_step‚2449 % _timestamp‚1.7844877648856406e+09 _runtime‚1241.460658673Ãã’42¯ - _wandbruntime‚1241  _runtime‚1241Ci¹qã'° ÝO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚189984716701184 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚189984716701184 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚187630811731456 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19492926 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚185662978588080 trainer/global_step‚2449 _runtime‚1241.461531824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2353564183552 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚185662995627440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚187630811731456 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19492649 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚149059577030144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24782275 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2426884080640 % _timestamp‚1.7844877648866732e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚189984358396416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13008960 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚189984358396416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43889651 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2353904969728 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚151486461110784 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19107376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19107376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43879506 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚188012668380560 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2353904969728 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19107374 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2349672753120 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2353564183552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6483689 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 _step‚196 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24772132 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚188012311417878 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19107374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24772132 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24782275 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2426750910464 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6483964 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚151486303516160 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚187630794212864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚149059552605696 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚187630794212864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0/ËfÃè'± âO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19107376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19107376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43879506 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚188012668380560 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2353904969728 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19107374 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2349672753120 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2353564183552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6483689 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 _step‚196 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24772132 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚188012311417878 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19107374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24772132 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24782275 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2426750910464 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6483964 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚151486303516160 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚187630794212864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚149059552605696 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚187630794212864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚189984716701184 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚189984716701184 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚187630811731456 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19492926 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚185662978588080 trainer/global_step‚2449 _runtime‚1241.461531824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2353564183552 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚185662995627440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚187630811731456 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19492649 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚149059577030144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24782275 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2426884080640 % _timestamp‚1.7844877648866732e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚189984358396416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13008960 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚189984358396416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43889651 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2353904969728 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚151486461110784 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088ĺâÒ2² - _wandbruntime‚1241  _runtime‚1241Z«u³ æM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24792164 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚188086665723594 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24782007 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19500219 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚151546296944128 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19114876 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2354494094848 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚190059480076800 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚188087022686388 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43907042 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187705003500544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2427706224128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2354834888192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43896883 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2354494094848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚187705003500544 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24792164 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19500503 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6486178 _runtime‚1241.823323653 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24782007 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 137357312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚197 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2350600976468 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43896883 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚187704985981952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚190059838388736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19114876 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2354834888192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43907042 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149118428938240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19114878 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13014041 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2350261053034 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6486460 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19114878 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚185736421709920 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚190059838388736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚2449 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13014043 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185736404670560 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088ϯ6sñ&´ ëM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149118453362688 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19114878 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13014041 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2350261053034 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6486460 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19114878 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚185736421709920 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚190059838388736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚2449 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13014043 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185736404670560 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24792164 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚188086665723594 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24782007 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19500219 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚151546296944128 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19114876 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2354494094848 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚190059480076800 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚188087022686388 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43907042 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187705003500544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2427706224128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2354834888192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43896883 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2354494094848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚187705003500544 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24792164 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19500503 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6486178 _runtime‚1241.823323653 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24782007 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 137357312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚197 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2350600976468 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43896883 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚187704985981952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚190059838388736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19114876 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2354834888192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43907042 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149118428938240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2474728995840 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚193740474672640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚191340400185856 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25277877 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25267734 % _timestamp‚1.7844877892417076e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚191340382667264 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚152003393657344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19482476 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193740474672640 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚191730415569568 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2400092005376 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚189334282712080 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13263012 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚191730058606886 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 _step‚200 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚191340382667264 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2395775894806 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚193740832977408 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19872952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2396115818128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44750208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19872657‰8Á âO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19872952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2396115818128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44750208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19872657 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 135267328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2474728995840 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚193740474672640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚191340400185856 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25277877 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25267734 % _timestamp‚1.7844877892417076e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚191340382667264 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚152003393657344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/a7°ctive.large_pool.allocated‚19482476 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193740474672640 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚191730415569568 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2400092005376 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚189334282712080 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13263012 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚191730058606886 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 _step‚200 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚191340382667264 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2395775894806 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚193740832977408 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145È"óí2 - _wandbruntime‚1266  _runtime‚1266À í&à çM E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159684608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2396704118042 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135260160 trainer/global_step‚2499 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚152062269989888 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2475690578432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚189407708794560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19489976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25277609 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13268093 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2397044041476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44767585 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25287766 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚154537936143872 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44767585 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2401021916672 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6612147 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚191414574436352 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44777744 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25277609 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6612430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚191414574436352 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 _runtime‚1266.175996847 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2475825838592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44777744 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19489978 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚152062245565440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19489978 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚193815596353024 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚193815954664960 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19880523 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13268091 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚201 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚191414591954944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2401362710016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚193815954664960 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25287766 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19880238 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚154538095828480 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19489976 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚189407725833920 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 % _timestamp‚1.7844877896008463e+09 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44767585 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2401021916672 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6612147 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚191414574436352 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44777744 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25277609 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6612430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚191414574436352 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 _runtime‚1266.175996847 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2475825838592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44777744 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19489978 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚152062245565440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19489978 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚193815596353024 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚193815954664960 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19880523 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13268091 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚201 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚191414591954944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2401362710016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚193815954664960 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25287766 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19880238 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚154538095828480 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19489976 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚189407725833920 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 % _timestamp‚1.7844877896008463e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159684608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2396704118042 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135260160 trainer/global_step‚2499 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚152062269989888 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2475690578432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚189407708794560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19489976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25277609 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13268093 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2397044041476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088ÉÙ2‡2Å - _wandbruntime‚1266  _runtime‚1266Œ¿&Æ € 2loss/SRqqt1/assignment_loss‚0.6184930970021801 /loss/FRt2/detection_loss‚0.5792654278811519 /loss/FBt1/detection_loss‚0.6253921809189851 epoch‚3 trainer/global_step‚2499 _runtime‚1266.176502027 1loss/SRqqt2/detection_loss‚0.9603487636097314 $ _timestamp‚1.784487789601814e+09 0loss/FRt1/assignment_loss‚1.3155165087293696 0loss/FRt2/assignment_loss‚2.5809645036689375 2loss/SRqqt2/assignment_loss‚0.6974151406811652 1loss/FBt1/assignment_loss‚0.12686149576124842 /loss/FRt1/detection_loss‚1.4229850297552291 /loss/FBt2/detection_loss‚0.7413951175080405 _step‚202 1loss/FBt2/assignment_loss‚0.14213358959630515 1loss/SRqqt1/detection_loss‚0.7296064108763057 %loss/total_loss‚10.54037726598865‰áþ Ç … $ _timestamp‚1.784487789601814e+09 0loss/FRt1/assignment_loss‚1.3155165087293696 0loss/FRt2/assignment_loss‚2.5809645036689375 2loss/SRqqt2/assignment_loss‚0.6974151406811652 1loss/FBt1/assignment_loss‚0.12686149576124842 /loss/FRt1/detection_loss‚1.4229850297552291 /loss/FBt2/detection_loss‚0.7413951175080405 _step‚202 1loss/FBt2/assignment_loss‚0.14213358959630515 1loss/SRqqt1/detection_loss‚0.7296064108763057 %loss/total_loss‚10.54037726598865 2loss/SRqqt1/assignment_loss‚0.6184930970021801 /loss/FRt2/detection_loss‚0.5792654278811519 /loss/FBt1/detection_loss‚0.6253921809189851 epoch‚3 trainer/global_step‚2499 _runtime‚1266.176502027 1loss/SRqqt2/detection_loss‚0.9603487636097314Êàóÿ2WÈ ‚(:Ì ö¾ôÒ€üâË disk.md0.in‚0 disk.dm-0.in‚ 4.3359375& disk.nvme0c0n1.out‚9544.2841796875$ memory_percent‚96.98786735534668* proc.memory.percent‚17.368102073669434 proc.cpu.threads‚17 network.sent‚ 3.217631e+06% disk./.usageGB‚1176.5439262390137% disk.nvme1n1.out‚3894.72216796875# disk.dm-0.out‚11335.41943359375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375$ disk.nvme0n1.out‚9544.2841796875) disk./.usagePercent‚76.63533145995855 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚3894.72216796875' proc.memory.availableMB‚ 493.5078125 cpu‚26.051330287252128 disk.md0.out‚ 37.32421875 disk.dm-1.out‚2023.958984375# proc.memory.rssMB‚ 2845.58984375 network.recv‚378868 disk.nvme1n1.in‚0ã$]4žÉ ‚(:“ ö¾ôÒÎÒ™Ì gpu.0.gpu‚66.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚54.0 gpu.0.powerWatts‚257.024' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.67466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|k ¼UÊ ‚(:Ê …¿ôÒ¬ÿÛË disk.dm-1.in‚0" disk.dm-1.out‚2024.12255859375% disk.nvme0n1.out‚9702.37646484375$ memory_percent‚98.49245548248291* proc.memory.percent‚17.368102073669434 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚9702.37646484375# proc.memory.rssMB‚ 2845.58984375( disk./.usagePercent‚76.5755242622747& disk.nvme1c1n1.out‚3968.9208984375 disk.md0.out‚ 37.72265625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.3359375 cpu‚26.048057866361386 network.sent‚ 3.269011e+06 network.recv‚382748% disk./.usageGB‚1175.6257362365723 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚3968.9208984375 disk.nvme0n1.in‚ 4.3359375( proc.memory.availableMB‚ 246.99609375 disk.dm-0.in‚ 4.3359375 disk.dm-0.out‚ 11566.6953125cðË ‚(:’ …¿ôÒÂ÷˜Ì gpu.0.gpu‚9.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚118.541' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.513666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕÛõ”2Ì - _wandbruntime‚1290  _runtime‚1290]÷.ÉÍ Ã ' lr-AdamW/pg1‚0.00021180926934161225 ' lr-AdamW/pg2‚0.00021180926934161225 trainer/global_step‚2549 % _timestamp‚1.7844878138044672e+09 _runtime‚1290.379478286 _step‚203‹aU(ÎÎ È _runtime‚1290.379478286 _step‚203 ' lr-AdamW/pg1‚0.00021180926934161225 ' lr-AdamW/pg2‚0.00021180926934161225 trainer/global_step‚2549 % _timestamp‚1.7844878138044672e+09Ë¡¬lÞ2Ï - _wandbruntime‚1290  _runtime‚1290]"%OªÐ ÜO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚195049971121664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2446960613376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚193005586836080 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2446619827200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19857576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13517060 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚154947185860096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 _runtime‚1290.380279588 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 _step‚204 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚197496590948864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25763336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13517062 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.784487813805425e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚197496949253632 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚195049971121664 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚195049988640256 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚195447805795894 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2442218959814 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19857576 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45620910 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 RDDeviceStatsMonitor.on_trai‡á_8n_batch_start/active_bytes.small_pool.peak‚ 538870784 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚197496590948864 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6735694 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6735965 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147108864 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚157469913597952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2446960613376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19857574 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122684416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚193005603875440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚197496949253632 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚197496590948864 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6735694 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6735965 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147108864 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚157469913597952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2446960613376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19857574 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122684416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚193005603875440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚197496949253632 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19857576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13517060 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚154947185860096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 _runtime‚1290.380279588 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 _step‚204 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚197496590948864 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25763336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13517062 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.784487813805425e+09Ì^„Ÿ82Ò - _wandbruntime‚1290  _runtime‚1290´Œ%í&Ó çM PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 120580096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2523690903552 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2447890531840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚1290.741819978 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45638287 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45648446 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20260609 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13522143 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19865076 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚195124162890752 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚197572070941184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 145004544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13522141 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19865078 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20260334 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚155006062192640 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚193079029957920 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2443487106484 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25773211 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚197571712629248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25783368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚155006086617088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚197572070941184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6738466 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚195124162890752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19865078 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25783368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚205 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚157529898100736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45638287 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2447549738496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2447549738496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45648446 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6738193 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2523811483648 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 òÄïŸÔ ìM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2523690903552 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2447890531840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚1290.741819978 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45638287 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45648446 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20260609 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13522143 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19865076 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚195124162890752 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚197572070941184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 145004544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13522141 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19865078 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20260334 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚155006062192640 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚193079029957920 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2443487106484 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25773211 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚197571712629248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25783368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚155006086617088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚197572070941184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 UIDeviceStatsMonitor.on_train_batch_end/in3Ì4€Sactive_split.small_pool.allocated‚6738466 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚195124162890752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19865078 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25783368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚205 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚157529898100736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45638287 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2447549738496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2447549738496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45648446 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6738193 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2523811483648 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 120580096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0Í¥Eõ"2Õ - _wandbruntime‚1290  _runtime‚1290h tÖÖ € 2loss/SRqqt2/assignment_loss‚0.6647023801218893 1loss/FBt1/assignment_loss‚0.10240467933862986 /loss/FRt1/detection_loss‚1.4230302929214065 0loss/SRqqt1/detection_loss‚0.681272378636767 /loss/FBt1/detection_loss‚0.5862337709606852 %loss/total_loss‚10.97494158691677 trainer/global_step‚2549 _runtime‚1290.742369779 0loss/FRt1/assignment_loss‚1.2267869528071338 /loss/FRt2/detection_loss‚0.5313322651214755 /loss/FRt2/assignment_loss‚2.645766177598168 2loss/SRqqt1/assignment_loss‚0.6655529065553666 2loss/FBt2/assignment_loss‚0.027246479981069376 1loss/SRqqt2/detection_loss‚0.9659091426292434 /loss/FBt2/detection_loss‚1.4547041602449358 epoch‚3 % _timestamp‚1.7844878141676836e+09 _step‚206GÐ4G × … % _timestamp‚1.7844878141676836e+09 _step‚206 2loss/SRqqt2/assignment_loss‚0.6647023801218893 1loss/FBt1/assignment_loss‚0.10240467933862986 /loss/FRt1/detection_loss‚1.4230302929214065 0loss/SRqqt1/detection_loss‚0.681272378636767 /loss/FBt1/detection_loss‚0.5862337709606852 %loss/total_loss‚10.97494158691677 trainer/global_step‚2549 _runtime‚1290.742369779 0loss/FRt1/assignment_loss‚1.2267869528071338 /loss/FRt2/detection_loss‚0.5313322651214755 /loss/FRt2/assignment_loss‚2.645766177598168 2loss/SRqqt1/assignment_loss‚0.6655529065553666 2loss/FBt2/assignment_loss‚0.027246479981069376 1loss/SRqqt2/detection_loss‚0.9659091426292434 /loss/FBt2/detection_loss‚1.4547041602449358 epoch‚3ÎãÌ]RØ ‚(:Ç ”¿ôÒ›ÝÖË% disk.nvme0c0n1.out‚9811.634765625# disk.nvme0n1.out‚9811.634765625$ disk.nvme1n1.out‚4009.6669921875# proc.memory.rssMB‚ 2845.58984375 network.sent‚ 3.271975e+06 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚4009.6669921875( proc.memory.availableMB‚ 246.92578125* proc.memory.percent‚17.368102073669434) disk./.usagePercent‚76.57474380540374$ disk./.usageGB‚1175.613754272461 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1n1.in‚0 proc.cpu.threads‚17 network.recv‚386251 disk.md0.in‚0# disk.dm-0.out‚11715.53173828125" disk.dm-1.out‚2024.32958984375 disk.nvme0n1.in‚ 4.3359375# memory_percent‚98.4928846359253 cpu‚26.044290632284756 disk.md0.out‚ 38.171875ŠH…ÖžÙ ‚(:“ ”¿ôÒ‘»Ì gpu.0.gpu‚64.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚253.969' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.65633333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰K9+2Ú - _wandbruntime‚1314  _runtime‚1314‰yh³ÉÛ Ã ' lr-AdamW/pg1‚0.00021174902664741535 ' lr-AdamW/pg2‚0.00021174902664741535 trainer/global_step‚2599 % _timestamp‚1.7844878384223564e+09 _runtime‚1314.998161475 _step‚207‰¢ªYÎÜ È % _timestamp‚1.7844878384223564e+09 _runtime‚1314.998161475 _step‚207 ' lr-AdamW/pg1‚0.00021174902664741535 ' lr-AdamW/pg2‚0.00021174902664741535 trainer/global_step‚2599ÏrŠS12Ý - _wandbruntime‚1314  _runtime‚1314ñÐ<á'Þ ÛO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201252707225088 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚196676907999440 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20232676 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198759559576064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚157891026911744 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2570860446720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13771112 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 135267328 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚201253065529856 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚201252707225088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159691776 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26258938 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚199165909947584 # _timestamp‚1.78448783842319e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26269081 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2599 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20232674 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚160461727666688 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚198759577094656 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20632891 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚208 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26269081 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46501757 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚198759577094656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2493488435200 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚300 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6861781 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚160461887358464 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚157891002487296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚198759559576064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2493147649024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20232676 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46491612 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚201253065529856 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2493488435200 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934?ØEæ'ß àO P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚196676907999440 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20232676 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198759559576064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚157891026911744 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2570860446720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13771112 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 135267328 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚201253065529856 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚201252707225088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159691776 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26258938 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚199165909947584 # _timestamp‚1.78448783842319e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26269081 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2599 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20232674 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚160461727666688 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚198759577094656 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20632891 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚208 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26269081 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46501757 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚198759577094656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2493488435200 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚300 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6861781 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚160461887358464 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚157891002487296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚198759559576064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2493147649024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20232676 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46491612 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚201253065529856 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2493488435200 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201252707225088ÐæÎ/2à - _wandbruntime‚1315  _runtime‚1315ŒÁUp$á çM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6864317 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 trainer/global_step‚2599 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚196750317042560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚160521562782208 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20640806 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46519148 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2489930171492 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2494418353664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13776193 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20240178 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20240176 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚298 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚199240264253412 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚198833751345152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2494077560320 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13776191 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚157949878819840 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚198833768863744 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46519148 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26268813 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 _step‚209 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20240178 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20240176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚198833751345152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133163008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20640508 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26268813 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26278970 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚157949903244288 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2571683962368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 % _timestamp‚1.7844878387864015e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6864613 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚296 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2489590248058 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ÐúÊ«ò&â ìM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚196750317042560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚160521562782208 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20640806 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46519148 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2489930171492 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2494418353664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13776193 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20240178 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20240176 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚298 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚199240264253412 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚198833751345152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2494077560320 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13776191 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚157949878819840 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚198833768863744 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46519148 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26268813 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 _step‚209 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20240178 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20240176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚198833751345152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133163008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20640508 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26268813 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26278970 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚157949903244288 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2571683962368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 % _timestamp‚1.7844878387864015e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6864613 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚296 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2489590248058 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6864317 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 trainer/global_step‚2599 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490Ñ’`We2ã - _wandbruntime‚1315  _runtime‚1315¶à¬Dä € 1loss/SRqqt2/detection_loss‚1.0022436956492213 trainer/global_step‚2599 0loss/FRt1/assignment_loss‚0.8506813432910655 1loss/FBt1/assignment_loss‚0.11661865902182798 &loss/total_loss‚11.522583618012941 _runtime‚1315.362156118 _step‚210 0loss/FRt2/assignment_loss‚2.7617108114834488 2loss/SRqqt2/assignment_loss‚0.7640650278611532 3loss/FBt2/assignment_loss‚0.0005418416366589034 /loss/FBt1/detection_loss‚0.6930368950288539 /loss/FBt2/detection_loss‚2.1320078068279793 " _timestamp‚1.7844878387874e+09 2loss/SRqqt1/assignment_loss‚0.6322942821581935 /loss/FRt1/detection_loss‚1.3256121617511358 .loss/FRt2/detection_loss‚0.596067298873697 1loss/SRqqt1/detection_loss‚0.6477037944297073 epoch‚3Ñ–k0 å … 0loss/FRt1/assignment_loss‚0.8506813432910655 1loss/FBt1/assignment_loss‚0.11661865902182798 &loss/total_loss‚11.522583618012941 _runtime‚1315.362156118 _step‚210 0loss/FRt2/assignment_loss‚2.7617108114834488 2loss/SRqqt2/assignment_loss‚0.7640650278611532 3loss/FBt2/assignment_loss‚0.0005418416366589034 /loss/FBt1/detection_loss‚0.6930368950288539 /loss/FBt2/detection_loss‚2.1320078068279793 " _timestamp‚1.7844878387874e+09 2loss/SRqqt1/assignment_loss‚0.6322942821581935 /loss/FRt1/detection_loss‚1.3256121617511358 .loss/FRt2/detection_loss‚0.596067298873697 1loss/SRqqt1/detection_loss‚0.6477037944297073 epoch‚3 1loss/SRqqt2/detection_loss‚1.0022436956492213 trainer/global_step‚2599Ò4ÛiNæ ‚(:à £¿ôÒåâìË% disk./.usageGB‚1175.6137504577637! disk.dm-0.out‚11736.740234375$ disk.nvme1n1.out‚4015.7412109375* proc.memory.percent‚17.368102073669434 network.sent‚ 3.335388e+06 disk.md0.out‚ 38.76953125 disk.rbd0.out‚0 disk.nvme0n1.in‚ 4.3359375& disk.nvme1c1n1.out‚4015.7412109375 disk.nvme1n1.in‚0$ memory_percent‚98.49307537078857( proc.memory.availableMB‚ 246.89453125) disk./.usagePercent‚76.57474355692972 disk.dm-0.in‚ 4.3359375! disk.dm-1.out‚2027.6298828125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375! disk.nvme0n1.out‚ 9831.3359375 disk.nvme1c1n1.in‚0# proc.memory.rssMB‚ 2845.58984375 network.recv‚390051 disk.md0.in‚0 disk.dm-1.in‚0# disk.nvme0c0n1.out‚ 9831.3359375 cpu‚26.04097418256713 proc.cpu.threads‚17Vvýžç ‚(:“ £¿ôÒñ€ÒÌ gpu.0.gpu‚90.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚203.398' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.79933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n#Œ“è ‚(:ˆ ²¿ôÒ“µÌ gpu.0.gpu‚95.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚264.663' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚88.221 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W#vŠVé ‚(:Ë ²¿ôÒ;‚Í network.recv‚394468) disk./.usagePercent‚76.57475076267608% disk.nvme0n1.out‚9832.15869140625 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚4016.13818359375 disk.nvme1n1.in‚0 disk.md0.out‚ 38.99609375! disk.dm-0.out‚11737.369140625 network.sent‚ 3.397728e+06 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 disk.rbd0.in‚0$ memory_percent‚98.49309921264648 cpu‚26.03806193900839 proc.cpu.threads‚17' disk.nvme0c0n1.out‚9832.15869140625 disk.dm-0.in‚ 4.3359375! disk.dm-1.out‚2027.7314453125 disk.rbd0.out‚0& proc.memory.availableMB‚ 246.890625* proc.memory.percent‚17.368102073669434% disk./.usageGB‚1175.6138610839844% disk.nvme1n1.out‚4016.13818359375 disk.md0.in‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2845.58984375 0šG2ê - _wandbruntime‚1339  _runtime‚1339 -BgÆë À trainer/global_step‚2649 $ _timestamp‚1.784487862596203e+09 _runtime‚1339.171218995 _step‚211 & lr-AdamW/pg1‚0.0002116870969692152 & lr-AdamW/pg2‚0.0002116870969692152¢hwÄËì Å & lr-AdamW/pg1‚0.0002116870969692152 & lr-AdamW/pg2‚0.0002116870969692152 trainer/global_step‚2649 $ _timestamp‚1.784487862596203e+09 _runtime‚1339.171218995 _step‚211Ó[ñð]2í - _wandbruntime‚1339  _runtime‚1339Ì*¥ã'î ÝO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚202469165549056 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2618847644672 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚160834843538944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 % _timestamp‚1.7844878625971668e+09 _step‚212 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚205008823501312 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26764683 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚2649 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2540016257024 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚163453537783296 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26754540 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21013990 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚200348195084080 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚205009181806080 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6988828 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14025160 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6988542 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚1339.172002286 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚205008823501312 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20607774 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21013702 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14025162 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚163453691183616 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47362314 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26764683 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2618718668800 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0°ÎŠâì ï âO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚202883657136592 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26764683 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚2649 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2540016257024 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚163453537783296 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26754540 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21013990 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚200348195084080 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚205009181806080 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6988828 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14025160 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6988542 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚1339.172002286 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚205008823501312 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20607774 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21013702 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14025162 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚163453691183616 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47362314 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26764683 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2618718668800 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚202469165549056 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2618847644672 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚160834843538944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 % _timestamp‚1.7844878625971668e+09 _step‚212 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚205008823501312 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26774572 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚202543339799552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚200421621166560 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2540946175488 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚202543357318144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2540946175488 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6991142 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20615276 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚202957654479626 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2540605382144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26764415 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21021383 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26764415 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚200421638205920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚160893695447040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20615278 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚163513530172928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2536373236500 _runtime‚1339.532824463 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47389850 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47389850 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚213 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚205084303493632 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 trainer/global_step‚2649 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26774572 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚202958011442420 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675埈®ò&ò ìM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚163513374682624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2540605382144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚160893719871488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6991426 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20615276 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21021669 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚200421621166560 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2540946175488 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚202543357318144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2540946175488 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6991142 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20615276 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚202957654479626 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2540605382144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26764415 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21021383 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26764415 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚200421638205920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚160893695447040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20615278 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚163513530172928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2536373236500 _runtime‚1339.532824463 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47389850 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47389850 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚213 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚205084303493632 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 trainer/global_step‚2649 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26774572 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚202958011442420 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚205083945181696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26774572 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚202543339799552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚208764939777536 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7115479 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚166445534005248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.¡lükq current‚10143 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 _step‚216 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20982876 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20982874 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21394966 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48233016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚208765298082304 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2586544078848 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2586544078848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2666873839104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2666746960384 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2699 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚163778635741696 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27250142 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2586203292672 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚206178736484864 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20982876 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚204019499208080 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7115754 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚48243161 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21394689 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚1363.323534028 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2586203292672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚204019516247440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚206178754003456 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27260285 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2581888078160 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚208764939777536 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚206178754003456 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20982874 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚208765298082304 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27260285 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27250142 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0R‡Ëè'ý âO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2586203292672 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚206178736484864 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20982876 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚204019499208080 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7115754 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚48243161 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21394689 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚1363.323534028 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2586203292672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚204019516247440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚206178754003456 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27260285 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2581888078160 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚208764939777536 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚206178754003456 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20982874 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚208765298082304 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27260285 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27250142 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚206601404325600 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚166445382702080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚208764939777536 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7115479 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚166445534005248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 _step‚216 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20982876 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20982874 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21394966 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48233016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚208765298082304 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2586544078848 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2586544078848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2666873839104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2666746960384 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2699 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚163778635741696 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27250142 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0Ø`BÈ2þ - _wandbruntime‚1363  _runtime‚1363½f”ì&ÿ æM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚206252945772544 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2582476378074 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚48250393 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚163837512074240 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21402561 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚48260552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚48250393 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.784487887104091e+09 _runtime‚1363.679186384 _step‚217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27270174 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚166505362484224 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 trainer/global_step‚2699 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20990378 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2587133203968 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20990376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚206252945772544 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14284291 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚166505221673984 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 116385792 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 140810240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚206252928253952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚204092925290560 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2667709599744 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20990378 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7118004 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚206675401668634 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚204092942329920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚208840419769856 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚206252928253952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48260552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21402295 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14284293 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27270174 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2582816301508 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚208840419769856 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2587133203968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚163837536498688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2667825985536 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7118268 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2587473997312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0$½¥ºf€ ëM EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21402295 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14284293 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27270174 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2582816301508 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚208840419769856 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2587133203968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚163837536498688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2667825985536 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7118268 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2587473997312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚206252945772544 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 XFDeCL‹viceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2582476378074 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚48250393 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚163837512074240 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21402561 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚48260552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚48250393 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.784487887104091e+09 _runtime‚1363.679186384 _step‚217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27270174 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚166505362484224 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 trainer/global_step‚2699 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20990378 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2587133203968 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20990376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚206252945772544 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14284291 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚166505221673984 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 116385792 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 140810240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚206252928253952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚204092925290560 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2667709599744 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20990378 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7118004 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚206675401668634 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚204092942329920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚208840419769856 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚206252928253952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48260552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚209888342457856 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2632731114496 _step‚220 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21357974 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚207690820371440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14533260 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2633071900672 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147108864 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122684416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7242865 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2628331143168 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚209888342457856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 % _timestamp‚1.7844879105649939e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2714923390976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21357976 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚166722452368896 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚1387.139841936 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚212521056053760 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7243146 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2627991219846 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27745744 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2714800706560 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21357976 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚166722452368896 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚1387.139841936 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚212521056053760 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7243146 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2627991219846 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27745744 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2714800706560 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚209888342457856 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2632731114496 _step‚220 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21357974 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚207690820371440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14533260 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2633071900672 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147108864 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122684416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7242865 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2628331143168 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚209888342457856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 % _timestamp‚1.7844879105649939e+09ÜA›âŒ2Ž - _wandbruntime‚1387  _runtime‚1387Aebà çM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚166781353125888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚207764246453920 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2629259366516 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49131254 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚210393505820436 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚209962516708352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2715885627392 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49131254 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDT’ eviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2633661025792 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21783946 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2633661025792 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚212596536046080 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2634001819136 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49121095 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚209962516708352 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚212596177734144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7245603 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27755619 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚209962534226944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14538341 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27755619 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2628919443082 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2715760852992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚1387.496342728 _step‚221 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚212596536046080 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚207764229414560 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21365476 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21783656 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7245315 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21365478 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210393148857642 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21365478 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27765776 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚169497238753280 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49131254 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2633661025792 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21783946 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2633661025792 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚212596536046080 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2634001819136 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49121095 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚209962516708352 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚212596177734144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7245603 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27755619 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚209962534226944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14538341 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27755619 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2628919443082 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2715760852992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚1387.496342728 _step‚221 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚212596536046080 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚207764229414560 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21365476 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21783656 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7245315 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21365478 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210393148857642ÝÙëЯ2‘ - _wandbruntime‚1387  _runtime‚13870ö$6 ’ ƒ /loss/FRt2/detection_loss‚0.6263222979726741 1loss/SRqqt1/detection_loss‚0.6503118376973429 &loss/total_loss‚10.501924937816494 _step‚222 /loss/FBt2/detection_loss‚0.7318881489725697 epoch‚3 _runtime‚1387.496830729 2loss/SRqqt1/assignment_loss‚0.6344692547185837 2loss/SRqqt2/assignment_loss‚0.6530163244280619 1loss/FBt2/assignment_loss‚0.19917013718797955 /loss/FRt1/detection_loss‚1.4587063269501035 0loss/FRt1/assignment_loss‚1.2047529855783838 2loss/FBt1/assignment_loss‚0.059523011978213664 1loss/SRqqt2/detection_loss‚0.9883669925575757 /loss/FBt1/detection_loss‚0.6529433786789397 trainer/global_step‚2749 % _timestamp‚1.7844879109221208e+09 0loss/FRt2/assignment_loss‚2.6424542410960643d¾»Ë“ ˆ 0loss/FRt1/assignment_loss‚1.2047529855783838 2loss/FBt1/assignment_loss‚0.059523011978213664 1loss/SRqqt2/detection_loss‚0.9883669925575757 /loss/FBt1/detection_loss‚0.6529433786789397 trainer/global_step‚2749 % _timestamp‚1.7844879109221208e+09 0loss/FRt2/assignment_loss‚2.6424542410960643 /loss/FRt2/detection_loss‚0.6263222979726741 1loss/SRqqt1/detection_loss‚0.6503118376973429 &loss/total_loss‚10.501924937816494 _step‚222 /loss/FBt2/detection_loss‚0.7318881489725697 epoch‚3 _runtime‚1387.496830729 2loss/SRqqt1/assignment_loss‚0.6344692547185837 2loss/SRqqt2/assignment_loss‚0.6530163244280619 1loss/FBt2/assignment_loss‚0.19917013718797955 /loss/FRt1/detection_loss‚1.4587063269501035Þ´™|Y” ‚(:Î î¿ôÒ—ÅË disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚4162.3544921875 disk.nvme0n1.in‚ 4.3359375# proc.memory.rssMB‚ 2845.58984375" disk.dm-1.out‚2029.04638671875 network.sent‚ 3.531233e+06$ disk.nvme1n1.out‚4162.3544921875 disk.md0.in‚0 disk.md0.out‚ 40.58203125 disk.nvme0c0n1.in‚ 4.3359375( disk.nvme0c0n1.out‚10866.52880859375& disk.nvme0n1.out‚10866.52880859375) disk./.usagePercent‚76.61783938647831 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚98.49522113800049( proc.memory.availableMB‚ 246.54296875* proc.memory.percent‚17.368102073669434 cpu‚26.02567428950877 disk.dm-1.in‚0 network.recv‚409781 disk.nvme1n1.in‚0 disk.dm-0.in‚ 4.3359375! disk.dm-0.out‚12913.205078125 proc.cpu.threads‚17% disk./.usageGB‚1176.2753791809082Ô Ó©œ• ‚(:‘ î¿ôÒîÓ£Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚54.0 gpu.0.powerWatts‚267.924' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.30799999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãé­N– ‚(:à ý¿ôÒîþÑË) disk./.usagePercent‚76.61783764716023 disk.nvme0n1.in‚ 4.3359375# memory_percent‚98.4954833984375! proc.memory.availableMB‚246.5 cpu‚26.0231174602305" disk.dm-1.out‚2029.28076171875 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚4167.439453125 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375" disk.dm-0.out‚12937.0458984375* proc.memory.percent‚17.368102073669434% disk./.usageGB‚1176.2753524780273 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 4.3359375# disk.nvme1n1.out‚4167.439453125 proc.cpu.threads‚17 network.sent‚ 3.595709e+06 network.recv‚413902 disk.rbd0.in‚0( disk.nvme0c0n1.out‚10887.36865234375& disk.nvme0n1.out‚10887.36865234375 disk.nvme1n1.in‚0 disk.md0.out‚ 41.4609375# proc.memory.rssMB‚ 2845.58984375à îž— ‚(:“ ý¿ôÒ»¡‡Ì gpu.0.gpu‚91.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚257.616' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.87199999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€ƒ3õ2˜ - _wandbruntime‚1410  _runtime‚1410Zâ˜`É™ à ' lr-AdamW/pg1‚0.00021149119608489638 ' lr-AdamW/pg2‚0.00021149119608489638 trainer/global_step‚2799 % _timestamp‚1.7844879343713028e+09 _runtime‚1410.947055068 _step‚223xŒÃΚ È ' lr-AdamW/pg1‚0.00021149119608489638 ' lr-AdamW/pg2‚0.00021149119608489638 trainer/global_step‚2799 % _timestamp‚1.7844879343713028e+09 _runtime‚1410.947055068 _step‚223ßôôª¿2› - _wandbruntime‚1410  _runtime‚1410¶µcDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 $ _timestamp‚1.784487934372067e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚216277172329984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49974420 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28251489 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2679599722496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚211362124495440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚169666293420544 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 _step‚224 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚213597930912256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚172429068447232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚213597930912256 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2674774208176 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚216277530634752 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚213597913393664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚216277530634752 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚211362107456080 _runtime‚1410.947117908 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21733076 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7370857 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21733074 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7370567 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2679258936320 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28241346 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚214036541740934 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2674434284854 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22158169 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚214036898703616 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28241346 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28251489 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14787310 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚172429219750400 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2679599722496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 $ _timestamp‚1.784487934372067e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚216277172329984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49974420 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28251489 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2679599722496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚211362124495440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚169666293420544 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 _step‚224 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚213597930912256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚172429068447232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚213597930912256 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2674774208176 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚216277530634752 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚213597913393664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚216277530634752 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚211362107456080 _runtime‚1410.947117908 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21733076 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7370857 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088à¸far2ž - _wandbruntime‚1411  _runtime‚1411Oœ…í&Ÿ çM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21740578 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14792391 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2680188847616 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2680188847616 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚213672122681344 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22165533 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚211435550577920 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 % _timestamp‚1.7844879347296538e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚214110896046650 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22165817 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚169725145328640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50001956 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21740578 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28261378 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28261378 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚213672122681344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2763758365184 _runtime‚1411.304804422 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28251221 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚216352294010368 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49991797 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚172489059184128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚213672105162752 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚172488903693824 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2763889431040 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21740576 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 Eò&  ìM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50001956 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49991797 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚213672105162752 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚172488903693824 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2763889431040 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21740576 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14792391 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2680188847616 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2680188847616 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚213672122681344 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22165533 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚211435550577920 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 % _timestamp‚1.7844879347296538e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚214110896046650 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22165817 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚169725145328640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50001956 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21740578 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28261378 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28261378 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚213672122681344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2763758365184 _runtime‚1411.304804422 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28251221 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚216352294010368 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49991797 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚172489059184128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 trainer/global_step‚2849 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2725786758144 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚217307501848064 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2810766733824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚220033288606208 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7498483 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚175421007854592 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚172610085623296 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚220033646910976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28747091 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15041360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2726127544320 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚220033288606208 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7498195 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2721217273184 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2720877349862 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22539845 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚217754288929942 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚217307519366656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15041362 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚217754645892624 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚217307519366656 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚217307501848064 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚217307501848064 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2810766733824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚220033288606208 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7498483 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚175421007854592 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚172610085623296 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚220033646910976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28747091 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15041360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2726127544320 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚220033288606208 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7498195 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2721217273184 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2720877349862 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22539845 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚217754288929942 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚217307519366656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15041362 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚217754645892624 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚217307519366656 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚217307501848064 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 trainer/global_step‚2849 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088ävH2¬ - _wandbruntime‚1434  _runtime‚1434»M›í&­ çM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50872658 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚220108410286592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2727057462784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚50862499 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚217381711135744 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22115676 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7500722 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚172668961955840 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 % _timestamp‚1.7844879581944633e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2727057462784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122677248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22115678 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22115678 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2722145496532 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚220108768598528 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚217381693617152 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28756980 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15046441 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2721805573098 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚217381711135744 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 trainer/global_step‚2849 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22115676 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28756980 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50872658 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚215106837662560 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 _step‚229 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚217828643235658 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚220108768598528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2811732240384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2726716669440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22547163 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚215106854701920 _runtime‚1434.769588036 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚50862499 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15046443 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚217829000198452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ÅÔ÷® ìM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50872658 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚220108410286592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2727057462784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚50862499 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚217381711135744 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22115676 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7500722 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚172668961955840 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 % _timestamp‚1.7844879581944633e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2727057462784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122677248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22115678 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22115678 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2722145496532 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚220108768598528 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚217381693617152 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28756980 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15046441 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2721805573098 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚217381711135744 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 trainer/global_step‚2849 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22115676 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28756980 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50872658 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚215106837662560 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 _step‚229 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚217828643235658 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚220108768598528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2811732240384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2726716669440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22547163 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚215106854701920 _runtime‚1434.769588036 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚50862499 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15046443 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚217829000198452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ålš2¯ - _wandbruntime‚1434  _runtime‚1434sâ³° ÿ _runtime‚1434.770082527 /loss/FRt2/assignment_loss‚2.595122528683608 epoch‚3 trainer/global_step‚2849 /loss/FBt2/detection_loss‚1.4163562849164009 0loss/FRt1/assignment_loss‚1.1277490985998209 1loss/SRqqt1/assignment_loss‚0.685549187553342 2loss/SRqqt2/assignment_loss‚0.7489921256832875 /loss/FRt1/detection_loss‚1.4742620610682617 .loss/FBt1/detection_loss‚4.476013851058989 _step‚230 /loss/FRt2/detection_loss‚0.5839037704068019 1loss/SRqqt2/detection_loss‚1.1008277099360437 &loss/total_loss‚14.966537569541535 % _timestamp‚1.7844879581953833e+09 1loss/FBt1/assignment_loss‚0.10469763823846141 1loss/FBt2/assignment_loss‚0.02119158113646904 1loss/SRqqt1/detection_loss‚0.6318717322600499OUfî ± „ /loss/FRt2/detection_loss‚0.5839037704068019 1loss/SRqqt2/detection_loss‚1.1008277099360437 &loss/total_loss‚14.966537569541535 % _timestamp‚1.7844879581953833e+09 1loss/FBt1/assignment_loss‚0.10469763823846141 1loss/FBt2/assignment_loss‚0.02119158113646904 1loss/SRqqt1/detection_loss‚0.6318717322600499 _runtime‚1434.770082527 /loss/FRt2/assignment_loss‚2.595122528683608 epoch‚3 trainer/global_step‚2849 /loss/FBt2/detection_loss‚1.4163562849164009 0loss/FRt1/assignment_loss‚1.1277490985998209 1loss/SRqqt1/assignment_loss‚0.685549187553342 2loss/SRqqt2/assignment_loss‚0.7489921256832875 /loss/FRt1/detection_loss‚1.4742620610682617 .loss/FBt1/detection_loss‚4.476013851058989 _step‚230æŒTùKU² ‚(:Ê ›ÀôÒ·¥þÌ" disk.dm-1.out‚2029.48193359375 proc.cpu.threads‚17& disk.nvme0c0n1.out‚10889.119140625 disk.nvme0n1.in‚ 4.3359375$ disk.nvme0n1.out‚10889.119140625' disk.nvme1c1n1.out‚4168.42529296875$ memory_percent‚98.49789142608643# proc.memory.rssMB‚ 2845.58984375) disk./.usagePercent‚76.61782522345963 disk.nvme1n1.in‚0 disk.md0.out‚ 42.0859375 disk.dm-1.in‚0* proc.memory.percent‚17.368102073669434 cpu‚26.017431470917174 network.recv‚421451$ disk./.usageGB‚1176.275161743164 disk.nvme0c0n1.in‚ 4.3359375% disk.nvme1n1.out‚4168.42529296875 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 disk.dm-0.out‚12938.20703125( proc.memory.availableMB‚ 246.10546875 network.sent‚ 3.660978e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0gß-ž³ ‚(:“ ›ÀôÒ„ÖºÍ gpu.0.gpu‚91.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚210.017' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚70.00566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÔiœÒY´ ‚(:Î ªÀôÒ¨â²Ë disk.nvme1c1n1.in‚0" disk.dm-0.out‚13231.5654296875" disk.dm-1.out‚2029.70849609375 disk.rbd0.out‚0& disk.nvme1c1n1.out‚4220.1708984375 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚4220.1708984375 disk.md0.out‚ 42.44140625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375( disk.nvme0c0n1.out‚11131.72314453125 disk.nvme0n1.in‚ 4.3359375 disk.dm-1.in‚0& disk.nvme0n1.out‚11131.72314453125$ memory_percent‚98.49801063537598' proc.memory.availableMB‚ 246.0859375* proc.memory.percent‚17.368102073669434 cpu‚26.01492983096462 proc.cpu.threads‚17 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375# proc.memory.rssMB‚ 2845.58984375 network.sent‚ 3.723423e+06 network.recv‚426003) disk./.usagePercent‚76.65143481218705% disk./.usageGB‚1176.7911529541016sY žµ ‚(:“ ªÀôÒ³ïúË gpu.0.gpu‚91.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚268.573' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.52433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò!}Î2¶ - _wandbruntime‚1458  _runtime‚1458–WmÉ· à ' lr-AdamW/pg1‚0.00021135217948051188 ' lr-AdamW/pg2‚0.00021135217948051188 trainer/global_step‚2899 % _timestamp‚1.7844879816723087e+09 _runtime‚1458.249399809 _step‚231IËG@θ È ' lr-AdamW/pg1‚0.00021135217948051188 ' lr-AdamW/pg2‚0.00021135217948051188 trainer/global_step‚2899 % _timestamp‚1.7844879816723087e+09 _runtime‚1458.249399809 _step‚231çðu2¹ - _wandbruntime‚1458  _runtime‚1458Ò_³ºã'º ÝO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2772314579968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15295412 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2858877001728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22483274 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 _step‚232 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22920530 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚218704715704080 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51725969 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7625118 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29232550 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚223789404882432 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚178412646082048 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15295410 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51715824 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22920233 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚221472393081632 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 trainer/global_step‚2899 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚175553926674944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚221017107821056 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2772655366144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚223789763187200 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2772655366144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚295 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚175553902250496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚1458.249493449 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2858743831552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29242693 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2772314579968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚221017107821056 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚223789404882432 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚221017090302464 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2767660338192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22483276 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22483274 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22483276 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚221017090302464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚297 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 % _timestamp‚1.7844879816729133e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29242693 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29232550 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2767320414870ñl0š"» âO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22483276 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22483274 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22483276 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚221017090302464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚297 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 % _timestamp‚1.7844879816729133e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29242693 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29232550 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2767320414870 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2772314579968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15295412 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2858877001728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22483274 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 _step‚232 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22920530 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚218704715704080 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51725969 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7625118 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29232550 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚223789404882432 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚178412646082048 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15295410 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51715824 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22920233 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚221472393081632 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 trainer/global_step‚2899 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚175553926674944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚221017107821056 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2772655366144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚223789763187200 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2772655366144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚295 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚175553902250496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚1458.2XLrŽN49493449 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2858743831552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29242693 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2772314579968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚221017107821056 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚223789404882432 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚221017090302464 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2767660338192è‹[ª2¼ - _wandbruntime‚1458  _runtime‚1458Ur'í&½ çM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7627621 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚221091282071552 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7627322 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚178472482549248 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51743360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29252582 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22927813 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 trainer/global_step‚2899 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚221546390424666 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2773585284608 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚175612778583040 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2859835032064 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29242425 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2773585284608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2859703966208 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚218778158825920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚223864884874752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚178472638039552 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22490776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22490776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22490778 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚223864526562816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚218778141786560 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚221091282071552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2768588561540 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7844879820314822e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51743360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22928114 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚299 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚233 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚221091299590144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2773244491264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22490778 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15300493 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29252582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51733201 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚221091299590144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚175612803007488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 _runtime‚1458.606635931 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784uVJüò&¾ ìM E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22490778 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15300493 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29252582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51733201 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚221091299590144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚175612803007488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 _runtime‚1458.606635931 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7627621 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚221091282071552 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7627322 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚178472482549248 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51743360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29252582 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22927813 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 trainer/global_step‚2899 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚221546390424666 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2773585284608 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚175612778583040 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2859835032064 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29242425 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2773585284608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2859703966208 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚218778158825920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚223864884874752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚178472638039552 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22490776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22490776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22490778 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚223864526562816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚218778141786560 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚221091282071552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2768588561540 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7844879820314822e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51743360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22928114 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚299 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚233 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚221091299590144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2773244491264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259éÿcÂà2¿ - _wandbruntime‚1458  _runtime‚1458¼@ÙßÀ € 1loss/SRqqt2/detection_loss‚1.0203671747512078 /loss/FBt1/detection_loss‚0.7031197831424523 .loss/FBt2/detection_loss‚1.321214220797022 0loss/FRt1/assignment_loss‚1.0456991551822648 1loss/FBt1/assignment_loss‚0.14420762558197095 trainer/global_step‚2899 _runtime‚1458.607131712 0loss/FBt2/assignment_loss‚0.1618724464640593 &loss/total_loss‚11.808825735065152 epoch‚3 % _timestamp‚1.7844879820324848e+09 2loss/SRqqt2/assignment_loss‚0.7155822868030842 1loss/SRqqt1/detection_loss‚0.6139080469968589 _step‚234 0loss/FRt2/assignment_loss‚3.0564178137262914 2loss/SRqqt1/assignment_loss‚0.7334212894691792 /loss/FRt1/detection_loss‚1.6062159268625074 /loss/FRt2/detection_loss‚0.6867999652882556’ƒs¸ Á … trainer/global_step‚2899 _runtime‚1458.607131712 0loss/FBt2/assignment_loss‚0.1618724464640593 &loss/total_loss‚11.808825735065152 epoch‚3 % _timestamp‚1.7844879820324848e+09 2loss/SRqqt2/assignment_loss‚0.7155822868030842 1loss/SRqqt1/detection_loss‚0.6139080469968589 _step‚234 0loss/FRt2/assignment_loss‚3.0564178137262914 2loss/SRqqt1/assignment_loss‚0.7334212894691792 /loss/FRt1/detection_loss‚1.6062159268625074 /loss/FRt2/detection_loss‚0.6867999652882556 1loss/SRqqt2/detection_loss‚1.0203671747512078 /loss/FBt1/detection_loss‚0.7031197831424523 .loss/FBt2/detection_loss‚1.321214220797022 0loss/FRt1/assignment_loss‚1.0456991551822648 1loss/FBt1/assignment_loss‚0.14420762558197095ê%¦tÜY ‚(:Î ¹ÀôÒæÍèË# proc.memory.rssMB‚ 2845.58984375) disk./.usagePercent‚76.65013057209929 disk.md0.in‚0" disk.dm-0.out‚13280.0224609375 disk.dm-1.in‚0( disk.nvme0c0n1.out‚11162.36865234375( proc.memory.availableMB‚ 246.29296875 cpu‚26.012067845435524 network.sent‚ 3.741263e+06 network.recv‚428429 disk.nvme1n1.in‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚11162.36865234375 proc.cpu.threads‚17% disk./.usageGB‚1176.7711296081543 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚4239.1044921875$ memory_percent‚98.49674701690674* proc.memory.percent‚17.368102073669434 disk.rbd0.out‚0$ disk.nvme1n1.out‚4239.1044921875 disk.md0.out‚ 42.921875 disk.dm-0.in‚ 4.3359375" disk.dm-1.out‚2029.82080078125 disk.nvme0n1.in‚ 4.3359375é:_žà ‚(:“ ¹ÀôÒ¹ žÌ gpu.0.gpu‚73.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚246.038' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚82.01266666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZÙÞÜ2Ä - _wandbruntime‚1482  _runtime‚1482É“‹¢ÇÅ Á trainer/global_step‚2949 $ _timestamp‚1.784488005639578e+09 _runtime‚ 1482.21640627 _step‚235 ' lr-AdamW/pg1‚0.00021128014967339154 ' lr-AdamW/pg2‚0.00021128014967339154¦# hÌÆ Æ ' lr-AdamW/pg2‚0.00021128014967339154 trainer/global_step‚2949 $ _timestamp‚1.784488005639578e+09 _runtime‚ 1482.21640627 _step‚235 ' lr-AdamW/pg1‚0.00021128014967339154ë.®G–2Ç - _wandbruntime‚1482  _runtime‚1482 ÊÈ ÜO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23301567 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2818842401792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚225189783307958 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚224726696275456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15549462 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 _step‚236 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚227545521158656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2814103403200 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23301859 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2906764855296 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227545521158656 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚227545879463424 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22858376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29728152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚224726678756864 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚52586526 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚224726678756864 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 _runtime‚ 1482.21647494 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7752397 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚181404483732992 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2819183187968 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚178497743302144 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚225190140270640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15549460 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 Ì*ç'É áO G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚227545879463424 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22858376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29728152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚224726678756864 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚52586526 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚224726678756864 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 _runtime‚ 1482.21647494 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7752397 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚181404483732992 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2819183187968 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚178497743302144 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚225190140270640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15549460 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23301567 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2818842401792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚225189783307958 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚224726696275456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15549462 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 _step‚236 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚227545521158656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2814103403200 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23301859 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2906764855296 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227545521158656 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0ì¬Z è2Ê - _wandbruntime‚1482  _runtime‚1482ábí&Ë çM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚224800870525952 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚52603903 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚52603903 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2815031626548 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚178556595210240 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2907855953408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52614062 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22865876 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚181464475588096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚222449445910560 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚178556619634688 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2820113106432 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚225264137613674 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚222449462949920 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚224800888044544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23309450 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚296 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 trainer/global_step‚2949 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚227621001150976 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2819772313088 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22865878 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29748184 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚181464318000640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22865876 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23309154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 % _timestamp‚1.7844880059977446e+09 _runtime‚1482.572839911 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2820113106432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2819772313088 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2907722790400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7754613 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29748184 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52614062 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088×ï·Àò&Ì ìM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚296 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 trainer/global_step‚2949 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚227621001150976 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2819772313088 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22865878 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29748184 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚181464318000640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22865876 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23309154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 % _timestamp‚1.7844880059977446e+09 _runtime‚1482.572839911 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2820113106432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2819772313088 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2907722790400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7754613 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29748184 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52614062 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚224800870525952 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚52603903 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚52603903 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2815031626548 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚178556595210240 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2907855953408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52614062 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22865876 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚181464475588096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚222449445910560 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚178556619634688 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2820113106432 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚225264137613674 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚222449462949920 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚224800888044544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23309450 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0íW›aò2Í - _wandbruntime‚1482  _runtime‚1482Áb!*ÑÎ ‚ 0loss/FRt1/assignment_loss‚1.0379217746081175 /loss/FRt2/detection_loss‚0.5961325856827433 /loss/FBt2/detection_loss‚1.8131690184275309 epoch‚3 _step‚238 4loss/FBt2/assignment_loäÇ^7ss‚0.00042168481274226604 .loss/FRt1/detection_loss‚1.409283167787793 1loss/SRqqt1/detection_loss‚0.7290766240744715 1loss/SRqqt2/detection_loss‚1.0965127116459572 /loss/FBt1/detection_loss‚0.5828164941089845 trainer/global_step‚2949 2loss/SRqqt2/assignment_loss‚0.5947578240089777 1loss/FBt1/assignment_loss‚0.10581192687943744 &loss/total_loss‚11.340602507415426 _runtime‚1482.573327232 .loss/FRt2/assignment_loss‚2.69761409480521 2loss/SRqqt1/assignment_loss‚0.6770846005734604 % _timestamp‚1.7844880059986904e+09iàêM Ï ‡ 2loss/SRqqt2/assignment_loss‚0.5947578240089777 1loss/FBt1/assignment_loss‚0.10581192687943744 &loss/total_loss‚11.340602507415426 _runtime‚1482.573327232 .loss/FRt2/assignment_loss‚2.69761409480521 2loss/SRqqt1/assignment_loss‚0.6770846005734604 % _timestamp‚1.7844880059986904e+09 0loss/FRt1/assignment_loss‚1.0379217746081175 /loss/FRt2/detection_loss‚0.5961325856827433 /loss/FBt2/detection_loss‚1.8131690184275309 epoch‚3 _step‚238 4loss/FBt2/assignment_loss‚0.00042168481274226604 .loss/FRt1/detection_loss‚1.409283167787793 1loss/SRqqt1/detection_loss‚0.7290766240744715 1loss/SRqqt2/detection_loss‚1.0965127116459572 /loss/FBt1/detection_loss‚0.5828164941089845 trainer/global_step‚2949î™:ARÐ ‚(:Ç ÈÀôÒľË disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375' disk.nvme0c0n1.out‚11291.0888671875 disk.nvme1n1.in‚0# proc.memory.rssMB‚ 2845.58984375 cpu‚26.00979088408565 network.sent‚ 3.7897e+06) disk./.usagePercent‚76.64884546451044 disk.dm-0.in‚ 4.3359375* proc.memory.percent‚17.368102073669434 disk.nvme1c1n1.in‚0% disk./.usageGB‚1176.7513999938965 disk.nvme0n1.in‚ 4.3359375% disk.nvme0n1.out‚11291.0888671875# disk.dm-0.out‚13440.27587890625 proc.cpu.threads‚17 network.recv‚432384 disk.rbd0.out‚0& disk.nvme1c1n1.out‚4271.4033203125$ disk.nvme1n1.out‚4271.4033203125 disk.md0.in‚0 disk.md0.out‚43.21875$ memory_percent‚98.49710464477539& proc.memory.availableMB‚ 246.234375 disk.dm-1.in‚0" disk.dm-1.out‚2029.93408203125»d½´žÑ ‚(:“ ÈÀôÒûƤÌ gpu.0.gpu‚74.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚54.0 gpu.0.powerWatts‚147.722' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚49.24066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U’LLYÒ ‚(:Î ×ÀôÒ•€¶Ë* proc.memory.percent‚17.368102073669434 network.recv‚436568! disk.dm-1.out‚2030.1123046875 disk.rbd0.out‚0# memory_percent‚98.4971284866333 cpu‚26.007053985253144 proc.cpu.threads‚17 network.sent‚ 3.85212e+06) disk./.usagePercent‚76.64886111837318% disk./.usageGB‚1176.7516403198242 disk.dm-0.in‚ 4.3359375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0# disk.dm-0.out‚13441.10009765625 disk.dm-1.in‚0' disk.nvme0c0n1.out‚11292.1826171875% disk.nvme0n1.out‚11292.1826171875 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 43.51171875( proc.memory.availableMB‚ 246.23046875' disk.nvme1c1n1.out‚4271.96044921875% disk.nvme1n1.out‚4271.96044921875# proc.memory.rssMB‚ 2845.58984375((äÖÓ ‚(:’ ×ÀôÒ©‰÷Ë gpu.0.gpu‚11.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚119.044' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.68133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î§ ö2Ô - _wandbruntime‚1506  _runtime‚1506‚Q½`ÈÕ Â trainer/global_step‚2999 $ _timestamp‚1.784488029604465e+09 _runtime‚1506.179439634 _step‚239 ' lr-AdamW/pg1‚0.00021120644039228432 ' lr-AdamW/pg2‚0.00021120644039228432]Ù‰ÏÍÖ Ç $ _timestamp‚1.784488029604465e+09 _runtime‚1506.179439634 _step‚239 ' lr-AdamW/pg1‚0.00021120644039228432 ' lr-AdamW/pg2‚0.00021120644039228432 trainer/global_step‚2999ïšÐ¹¼2× - _wandbruntime‚1506  _runtime‚1506QÓZ¾ã'Ø ÝO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53457228 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2865711009792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚226047323952080 % _timestamp‚1.7844880296053295e+09 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30223754 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23682630 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2954676582400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚228907887459648 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚240 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚228436284729856 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2865370223616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 trainer/global_step‚2999 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚231301637434880 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 _runtime‚1506.180182855 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30233897 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23682341 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15803512 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚181441559929344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30233897 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚231301995739648 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚228436284729856 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚181441535504896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚228436267211264 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7879118 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7878831 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚228907530496966 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚231301995739648 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23233476 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23233474 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2865370223616 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚184396212087296 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53467373 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 Ÿ_è'Ù âO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚228436267211264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2865711009792 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23233476 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2860546468208 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23233474 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30223754 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23233476 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23233474 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2865370223616 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚184396212087296 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53467373 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53457228 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2865711009792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚226047323952080 % _timestamp‚1.7844880296053295e+09 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30223754 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23682630 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2954676582400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚228907887459648 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚240 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚228436284729856 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2865370223616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 trainer/global_step‚2999 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚231301637434880 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 _runtime‚1506.180182855 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30233897 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23682341 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15803512 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚181441559929344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30233897 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚231301995739648 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚228436284729856 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚181441535504896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚228436267211264 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7879118 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7878831 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚228907530496966 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚231301995739648 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚226120750034560 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚231377117427200 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚228981884802682 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1506.537494742 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚231376759115264 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2866300134912 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚231377117427200 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚184456044691968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚228510476498944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133163008 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7881368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚181500411837440 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30243786 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2866640928256 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚181500436261888 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 trainer/global_step‚2999 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53484764 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30233629 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2861134768122 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2955766017536 % _timestamp‚1.7844880299624033e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23689959 VIDeviceStatsMonitor.on_train_batch_end/inactiv11íoe_split.large_pool.allocated‚15808593 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184456202279424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2866640928256 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2955632854528 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53484764 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157¯ýò&Ü ìM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30233629 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2861134768122 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2955766017536 % _timestamp‚1.7844880299624033e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23689959 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15808593 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184456202279424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2866640928256 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2955632854528 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53484764 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚226120750034560 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚231377117427200 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚228981884802682 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1506.537494742 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚231376759115264 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2866300134912 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚231377117427200 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚184456044691968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚228510476498944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133163008 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7881368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚181500411837440 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30243786 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2866640928256 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚181500436261888 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 trainer/global_step‚2999 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53484764ñãåŸØ2Ý - _wandbruntime‚1506  _runtime‚1506:VMbÞ ÿ 1loss/FBt1/assignment_loss‚0.05240814868726727 /loss/FRt2/detection_loss‚0.6221726602955422 /loss/FBt2/detection_loss‚0.9890764860006479 &loss/total_loss‚11.176125336518295 1loss/SRqqt1/detection_loss‚0.6745316375628225 0loss/SRqqt2/detection_loss‚0.961998496706096 _step‚242 /loss/FRt2/assignment_loss‚2.697169794604696 trainer/global_step‚2999 _runtime‚1506.537976523 2loss/SRqqt2/assignment_loss‚0.7154636855728433 2loss/FBt2/assignment_loss‚0.002232930954523209 .loss/FRt1/detection_loss‚1.450703149911499 /loss/FBt1/detection_loss‚1.3421001564853268 epoch‚3 $ _timestamp‚1.784488029963315e+09 0loss/FRt1/assignment_loss‚1.0347966768123484 2loss/SRqqt1/assignment_loss‚0.6334715129246831þƒ| ß „ 1loss/SRqqt1/detection_loss‚0.6745316375628225 0loss/SRqqt2/detection_loss‚0.961998496706096 _step‚242 /loss/FRt2/assignment_loss‚2.697169794604696 trainer/global_step‚2999 _runtime‚1506.537976523 2loss/SRqqt2/assignment_loss‚0.7154636855728433 2loss/FBt2/assignment_loss‚0.002232930954523209 .loss/FRt1/detection_loss‚1.450703149911499 /loss/FBt1/detection_loss‚1.3421001564853268 epoch‚3 $ _timestamp‚1.784488029963315e+09 0loss/FRt1/assignment_loss‚1.0347966768123484 2loss/SRqqt1/assignment_loss‚0.6334715129246831 1loss/FBt1/assignment_loss‚0.05240814868726727 /loss/FRt2/detection_loss‚0.6221726602955422 /loss/FBt2/detection_loss‚0.9890764860006479 &loss/total_loss‚11.176125336518295òˆÝ)gZà ‚(:Ï æÀôÒŸÍÔË network.sent‚ 3.858031e+06 disk.nvme0n1.in‚ 4.3359375' disk.nvme1c1n1.out‚4276.09423828125% disk.nvme1n1.out‚4276.09423828125 disk.nvme0c0n1.in‚ 4.3359375 cpu‚26.004523548403434 network.recv‚439845( disk./.usagePercent‚76.6489038559032$ disk./.usageGB‚1176.752296447754 disk.md0.in‚0 disk.md0.out‚ 43.89453125 disk.dm-0.in‚ 4.3359375 disk.rbd0.in‚0* proc.memory.percent‚17.368102073669434& disk.nvme0n1.out‚11308.99267578125 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚11308.99267578125$ memory_percent‚98.49731922149658( proc.memory.availableMB‚ 246.19921875" disk.dm-0.out‚13461.1123046875 disk.dm-1.in‚0! disk.dm-1.out‚2030.2216796875# proc.memory.rssMB‚ 2845.58984375 proc.cpu.threads‚17k,žá ‚(:“ æÀôÒ‰ÞµÌ gpu.0.gpu‚63.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚251.057' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚83.68566666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%q+2â - _wandbruntime‚1529  _runtime‚1529yaŽßÄã ¾ % lr-AdamW/pg1‚0.000211131052817794 % lr-AdamW/pg2‚0.000211131052817794 trainer/global_step‚3049 $ _timestamp‚1.784488053261444e+09 _runtime‚1529.836431252 _step‚2431˜ÜÉä à % lr-AdamW/pg1‚0.000211131052817794 % lr-AdamW/pg2‚0.000211131052817794 trainer/global_step‚3049 $ _timestamp‚1.784488053261444e+09 _runtime‚1529.836431252 _step‚243ótä12å - _wandbruntime‚1529  _runtime‚1529Aħã'æ ÝO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚229718628076080 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2912238831616 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54338075 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24063790 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚187388210779648 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30719356 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚232145873184256 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 _step‚244 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7844880532623289e+09 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8006500 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚232625277685974 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30729499 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2912238831616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 145011712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3002834223104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 120587264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 trainer/global_step‚3049 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚235058112015872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16057560 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184385376556544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30719356 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16057562 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30729499 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54327930 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2911898045440 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚235057753711104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235057753711104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚232145873184256 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23608576 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23608576 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚232145855665664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2906989533216 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚235058112015872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23608574 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54338075 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24063790 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚187388210779648 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30719356 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚232145873184256 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 _step‚244 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7844880532623289e+09 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8006500 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚232625277685974 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30729499 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2912238831616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 145011712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3002834223104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 120587264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 trainer/global_step‚3049 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚235058112015872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16057560 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184385376556544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30719356 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16057562 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30729499 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes± Ý?± .small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54327930 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2911898045440 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚235057753711104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235057753711104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚232145873184256 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23608576 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23608576 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚232145855665664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2906989533216 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚235058112015872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23608574 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 _step‚245 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23616076 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚232699631991690 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30729231 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30729231 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23616078 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30739388 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122677248 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8008808 _runtime‚1530.194924515 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54345307 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚235133233703424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2907917756564 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54355466 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2912827956736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23616076 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54345307 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚229792071197920 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚232220047434752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16062641 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 147101696 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3003799634432 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚187448052523520 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚232220064953344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚187447905421824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚232220047434752 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2907577833130 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚229792054158560 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚184444228464640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23616078 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚235132875391488 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54355466 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2913168750080 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24071449 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚232699988954484 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30739388 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚184444252889088 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3003676957184 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8009100 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368õNë"ò&ì ìM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30729231 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30729231 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23616078 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30739388 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122677248 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8008808 _runtime‚1530.194924515 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54345307 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚235133233703424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2907917756564 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54355466 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2912827956736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23616076 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54345307 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚229792071197920 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚232220047434752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16062641 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 147101696 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3003799634432 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚187448052523520 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚232220064953344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚187447905421824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚232220047434752 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2907577833130 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚229792054158560 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚184444228464640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23616078 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚235132875391488 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54355466 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2913168750080 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24071449 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚232699988954484 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30739388 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚184444252889088 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3003676957184 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8009100 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24071743 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 _step‚245 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23616076 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚232699631991690 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375õån]—2í - _wandbruntime‚1530  _runtime‚1530cö–î ÿ 0loss/FRt2/assignment_loss‚2.4997152186990754 1loss/SRqqt1/detection_loss‚0.6849984765209467 epoch‚3 % _timestamp‚1.7844880536207502e+09 1loss/FBt2/assignment_loss‚0.12644545076166955 .loss/FBt2/detection_loss‚2.161853318413099 2loss/SRqqt2/assignment_loss‚0.5643061930555632 1loss/FBt1/assignment_loss‚0.05776731566832657 0loss/SRqqt2/detection_loss‚1.014418768096948 /loss/FBt1/detection_loss‚0.6992711685889087 trainer/global_step‚3049 _runtime‚1530.195372056 0loss/FRt1/assignment_loss‚1.1280357069969178 2loss/SRqqt1/assignment_loss‚0.5425766138947421 .loss/FRt1/detection_loss‚1.576757903907448 /loss/FRt2/detection_loss‚0.5858323256951091 &loss/total_loss‚11.641978460298754 _step‚246è¢Om ï „ 0loss/FRt1/assignment_loss‚1.1280357069969178 2loss/SRqqt1/assignment_loss‚0.5425766138947421 .loss/FRt1/detection_loss‚1.576757903907448 /loss/FRt2/detection_loss‚0.5858323256951091 &loss/total_loss‚11.641978460298754 _step‚246 0loss/FRt2/assignment_loss‚2.4997152186990754 1loss/SRqqt1/detection_loss‚0.6849984765209467 epoch‚3 % _timestamp‚1.7844880536207502e+09 1loss/FBt2/assignment_loss‚0.12644545076166955 .loss/FBt2/detection_loss‚2.161853318413099 2loss/SRqqt2/assignment_loss‚0.5643061930555632 1loss/FBt1/assignment_loss‚0.05776731566832657 0loss/SRqqt2/detection_loss‚1.014418768096948 /loss/FBt1/detection_loss‚0.6992711685889087 trainer/global_step‚3049 _runtime‚1530.195372056öâ%Rþžð ‚(:“ „ÁôÒºÖŒÌ gpu.0.gpu‚91.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚266.975' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.99166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïUñ ‚(:Ê „ÁôÒÖæÌ disk.rbd0.in‚0 network.recv‚448353$ disk./.usageGB‚1174.020378112793 disk.md0.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 4.3359375* proc.memory.percent‚17.368102073669434 disk.nvme1n1.in‚0 disk.md0.out‚ 44.578125# disk.dm-0.out‚13464.12158203125 disk.nvme0c0n1.in‚ 4.3359375& disk.nvme1c1n1.out‚4277.4912109375$ memory_percent‚98.49686622619629 cpu‚25.999906457271425 disk.rbd0.out‚0 network.sent‚ 3.971009e+06( disk./.usagePercent‚76.4709577015335$ disk.nvme1n1.out‚4277.4912109375 disk.dm-0.in‚ 4.3359375' proc.memory.availableMB‚ 246.2734375' disk.nvme0c0n1.out‚11312.2587890625% disk.nvme0n1.out‚11312.2587890625 disk.nvme1c1n1.in‚0# proc.memory.rssMB‚ 2845.58984375 proc.cpu.threads‚17" disk.dm-1.out‚2030.39306640625M ¨"2ò - _wandbruntime‚1553  _runtime‚1553Œ!DÄó ¾ & lr-AdamW/pg1‚0.0002110539881574056 & lr-AdamW/pg2‚0.0002110539881574056 trainer/global_step‚3099 # _timestamp‚1.78448807719223e+09 _runtime‚ 1553.76722073 _step‚247_ÀMvÉô à _step‚247 & lr-AdamW/pg1‚0.0002110539881574056 & lr-AdamW/pg2‚0.0002110539881574056 trainer/global_step‚3099 # _timestamp‚1.78448807719223e+09 _runtime‚ 1553.76722073÷¶aÂ82õ - _wandbruntime‚1553  _runtime‚1553=ÓÌ ´ö ÝO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2958425867264 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238813869987328 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2958766653440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2958766653440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23983674 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚238814228292096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚190380089199104 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3050767039488 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31214958 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2958425867264 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31225101 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚233389932200080 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31214958 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚235855461638656 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 % _timestamp‚1.7844880771931365e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚238814228292096 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3050896015360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 _step‚248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚55208777 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚238813869987328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2953092674902 trainer/global_step‚3099 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23983676 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8134711 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2953432598224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259¹ “Yè'÷ âO % _timestamp‚1.7844880771931365e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚238814228292096 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3050896015360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 _step‚248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚55208777 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚238813869987328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2953092674902 trainer/global_step‚3099 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23983676 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8134711 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2953432598224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2958425867264 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238813869987328 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2958766653440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2958766653440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23983674 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚238814228292096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚190380089199104 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3050767039488 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31214958 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2958425867264 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31225101 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚233389932200080 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31214958 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚235855461638656 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934ø¦®92ø - _wandbruntime‚1554  _runtime‚1554v˜ƒ†í&ù çM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31224833 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚187388045091840 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚238889349979648 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2959696571904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1554.125985655 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚238889349979648 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚190439927645696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31234990 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚3099 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55226168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚187388069516288 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚236417379180698 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚238888991667712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16316691 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚236417736143492 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23991178 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23991176 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24453955 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8137539 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2954360821572 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55226168 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2959355778560 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚190439776349696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23991178 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚235929653407744 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚235929653407744 % _timestamp‚1.7844880775508113e+09ÃÌ{]§ú ìM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚235929635889152 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55216009 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24454232 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55226168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚187388069516288 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚236417379180698 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚238888991667712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16316691 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚236417736143492 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23991178 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23991176 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24453955 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8137539 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2954360821572 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55226168 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2959355778560 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚190439776349696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23991178 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚235929653407744 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚235929653407744 % _timestamp‚1.7844880775508113e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31224833 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚187388045091840 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚238889349979648 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2959696571904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1554.125985655 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚238889349979648 C9DeviceStatsMq3ÞK onitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚190439927645696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31234990 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚3099 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚193371907004928 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8261961 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚193371759896064 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚240060772063990 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3005294475264 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24358776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56069334 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242569986263552 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8262254 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24358776 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3005294475264 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3004953689088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3098774509568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚239565032574464 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3004953689088 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31710560 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31720703 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚237061236324080 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147108864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 _step‚252 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56079479 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16565660 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2999875663232 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24827621 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31710560 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16565662ñ<è'‡ âO TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚3149 % _timestamp‚1.7844881018181946e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚239565032574464 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31720703 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚237061236324080 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147108864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 _step‚252 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56079479 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16565660 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2999875663232 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24827621 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31710560 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16565662 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚239565050093056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚190272985386496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚239565050093056 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚242569986263552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚193371907004928 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8261961 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚193371759896064 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚240060772063990 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3005294475264 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24358776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56069334 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242569986263552 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8262254 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24358776 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3005294475264 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3004953689088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3098774509568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚239565032574464 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3004953689088 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31710560 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3006224393728 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3099864430080 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 % _timestamp‚1.7844881021797807e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3005883600384 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3006224393728 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚237134662406560 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚295 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚193431750573568 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚190331861719040 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚253 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚242645466255872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚239639224343552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56086711 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3099733364224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚242645107943936 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24366278 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31730592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/r>êt Ãequested_bytes.small_pool.freed‚ 3000463963146 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56096870 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193431595083264 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _runtime‚1578.754957919 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚239639241862144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16570741 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚240135483332500 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚240135126369706 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24835255 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8264514 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56086711 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31720435 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8264809 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24366276 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24835552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚190331886143488 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56096870 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0a˜Aò&Š ìM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚190331886143488 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56096870 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3006224393728 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3099864430080 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 % _timestamp‚1.7844881021797807e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3005883600384 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3006224393728 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚237134662406560 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚295 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚193431750573568 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚190331861719040 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚253 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚242645466255872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚239639224343552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56086711 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3099733364224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚242645107943936 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24366278 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31730592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3000463963146 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56096870 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193431595083264 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _runtime‚1578.754957919 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚239639241862144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16570741 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚240135483332500 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚240135126369706 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24835255 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8264514 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56086711 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31720435 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8264809 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24366276 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24835552ýª\²2‹ - _wandbruntime‚1578  _runtime‚1578Óýì Œ ƒ 0loss/FBt1/detection_loss‚0.42435812163554754 0loss/FRt2/assignment_loss‚2.3620061462529227 /loss/FRt2/detection_loss‚0.5488805814125376 % _timestamp‚1.7844881021812136e+09 _step‚254 %loss/total_loss‚9.916018389832686 2loss/SRqqt2/assignment_loss‚0.6476103309033947 2loss/FBt1/assignment_loss‚0.061655664772100376 /loss/FRt1/detection_loss‚1.4041188594529508 trainer/global_step‚3149 2loss/SRqqt1/assignment_loss‚0.6398386053905448 epoch‚3 _runtime‚ 1578.75591939 /loss/FBt2/detection_loss‚1.0159845285945468 0loss/FRt1/assignment_loss‚1.0420920954445216 2loss/FBt2/assignment_loss‚0.014332521906605203 1loss/SRqqt1/detection_loss‚0.6168504267319127 1loss/SRqqt2/detection_loss‚1.1382905073351028„qWT ˆ %loss/total_loss‚9.916018389832686 2loss/SRqqt2/assignment_loss‚0.6476103309033947 2loss/FBt1/assignment_loss‚0.061655664772100376 /loss/FRt1/detection_loss‚1.4041188594529508 trainer/global_step‚3149 2loss/SRqqt1/assignment_loss‚0.6398386053905448 epoch‚3 _runtime‚ 1578.75591939 /loss/FBt2/detection_loss‚1.0159845285945468 0loss/FRt1/assignment_loss‚1.0420920954445216 2loss/FBt2/assignment_loss‚0.014332521906605203 1loss/SRqqt1/detection_loss‚0.6168504267319127 1loss/SRqqt2/detection_loss‚1.1382905073351028 0loss/FBt1/detection_loss‚0.42435812163554754 0loss/FRt2/assignment_loss‚2.3620061462529227 /loss/FRt2/detection_loss‚0.5488805814125376 % _timestamp‚1.7844881021812136e+09 _step‚254þ°ljSŽ ‚(:È ±ÁôÒà®åË disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375$ memory_percent‚98.50192070007324 proc.memory.rssMB‚ 2845.59375 network.sent‚ 4.090598e+06( disk./.usagePercent‚76.6548702138737% disk.nvme1n1.out‚4285.46923828125 disk.md0.in‚0 disk.md0.out‚ 45.5234375 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0 disk.rbd0.in‚0 network.recv‚459702& disk.nvme0n1.out‚13232.35107421875' proc.memory.availableMB‚ 245.4453125 disk.dm-1.out‚2030.810546875$ disk./.usageGB‚1176.843894958496 disk.rbd0.out‚0( disk.nvme0c0n1.out‚13232.35107421875* proc.memory.percent‚17.368125915527344 cpu‚26.00243712618332 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚4285.46923828125 disk.nvme1n1.in‚0" disk.dm-0.out‚15389.7236328125¾¥ú’ ‚(:‡ ±ÁôÒžÁÌ gpu.0.gpu‚22.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚119.172' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.724 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ðk2 - _wandbruntime‚1602  _runtime‚1602ŽV$Ç‘ Á _step‚255 & lr-AdamW/pg1‚0.0002108948325431645 & lr-AdamW/pg2‚0.0002108948325431645 trainer/global_step‚3199 % _timestamp‚1.7844881258349562e+09 _runtime‚1602.411416826îä}ÕÌ’ Æ _step‚255 & lr-AdamW/pg1‚0.0002108948325431645 & lr-AdamW/pg2‚0.0002108948325431645 trainer/global_step‚3199 % _timestamp‚1.7844881258349562e+09 _runtime‚1602.411416826ÿo§!2“ - _wandbruntime‚1602  _runtime‚1602\Wã'” ÝO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3051822297088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16819712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚193216826438144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚193216802013696 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 _runtime‚1602.411475356 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3051481510912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚243274638547456 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24733876 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246326102539776 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56940036 trainer/global_step‚3199 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25207850 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚196363728740864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚246326102539776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122684416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240732540448080 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚240732557487440 % _timestamp‚1.7844881258356304e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32216305 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3146902302720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32216305 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚243778519252998 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚256 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3051822297088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚243274638547456 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32206162 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚243778876215680 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3046318728240 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3045978804918 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚246326460844544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088çç¾sè• âO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚193216802013696 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 _runtime‚1602.411475356 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3051481510912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚243274638547456 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24733876 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246326102539776 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56940036 trainer/global_step‚3199 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25207850 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚196363728740864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚246326102539776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122684416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240732540448080 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚240732557487440 % _timestamp‚1.7844881258356304e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 HBD6D¬eviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32216305 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3146902302720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32216305 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚243778519252998 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚256 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3051822297088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚243274638547456 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32206162 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚243778876215680 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3046318728240 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3045978804918 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚246326460844544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3051822297088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16819712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚193216826438144€ó>&þ2– - _wandbruntime‚1602  _runtime‚1602k†Ïví&— çM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24741378 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56957413 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32216037 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚196423565350400 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3046907028154 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚193275702770688 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚240805966530560 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3047246951588 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚243348812797952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24741378 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚1602.768329142 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3147741999616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56967572 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8390628 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25215694 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24741376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚243348830316544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚196423420345856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 120580096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚243852873558714 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 % _timestamp‚1.7844881261931672e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 trainer/global_step‚3199 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3052411422208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25215419 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚246401582532096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56967572 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32216037 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚240805983569920 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16824791 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 145004544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 _step‚257 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3052752215552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32226194 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16824793 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56957413 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚246401224220160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3052752215552 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3147862579712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720÷jb™ò&˜ ìM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24741376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚243348830316544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚196423420345856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 120580096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚243852873558714 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 % _timestamp‚1.7844881261931672e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 trainer/global_step‚3199 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3052411422208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25215419 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚246401582532096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56967572 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32216037 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚240805983569920 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16824791 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 145004544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 _step‚257 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3052752215552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32226194 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16824793 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56957413 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚246401224220160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3052752215552 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3147862579712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24741378 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56957413 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32216037 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚196423565350400 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3046907028154 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚193275702770688 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚240805966530560 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3047246951588 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚243348812797952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24741378 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚1602.768329142 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3147741999616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56967572 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8390628 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25215694’«E2™ - _wandbruntime‚1602  _runtime‚1602éÓ‹š € 0loss/FRt2/assignment_loss‚2.5885891016477194 2loss/FBt2/assignment_loss‚0.055105472531189054 .loss/FRt1/detection_loss‚1.383799290311315 /loss/SRqqt2/detection_loss‚1.21149685221496 .loss/FBt1/detection_loss‚0.532479765318955 trainer/global_step‚3199 _runtime‚1602.768787513 2loss/SRqqt2/assignment_loss‚0.6380417661871283 /loss/FRt2/detection_loss‚0.6035960263738022 &loss/total_loss‚10.680507067626742 0loss/FRt1/assignment_loss‚0.9971070853497485 2loss/SRqqt1/assignment_loss‚0.6576156960914029 _step‚258 2loss/FBt1/assignment_loss‚0.006394696787659863 1loss/SRqqt1/detection_loss‚0.6434734453983744 /loss/FBt2/detection_loss‚1.3628078694144885 epoch‚3 % _timestamp‚1.7844881261941466e+09ì¯/² › … 2loss/FBt1/assignment_loss‚0.006394696787659863 1loss/SRqqt1/detection_loss‚0.6434734453983744 /loss/FBt2/detection_loss‚1.3628078694144885 epoch‚3 % _timestamp‚1.7844881261941466e+09 0loss/FRt2/assignment_loss‚2.5885891016477194 2loss/FBt2/assignment_loss‚0.055105472531189054 .loss/FRt1/detection_loss‚1.383799290311315 /loss/SRqqt2/detection_loss‚1.21149685221496 .loss/FBt1/detection_loss‚0.532479765318955 trainer/global_step‚3199 _runtime‚1602.768787513 2loss/SRqqt2/assignment_loss‚0.6380417661871283 /loss/FRt2/detection_loss‚0.6035960263738022 &loss/total_loss‚10.680507067626742 0loss/FRt1/assignment_loss‚0.9971070853497485 2loss/SRqqt1/assignment_loss‚0.6576156960914029 _step‚258‚ðmôUQœ ‚(:Æ ÀÁôÒŸ’ÐË disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 network.sent‚ 4.154062e+06 disk.md0.out‚ 46.140625( disk.nvme0c0n1.out‚13240.05224609375 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0% disk.nvme1n1.out‚4287.48974609375 disk.dm-0.in‚ 4.3359375& disk.nvme0n1.out‚13240.05224609375' disk.nvme1c1n1.out‚4287.48974609375$ memory_percent‚98.50349426269531 proc.memory.rssMB‚ 2845.59375* proc.memory.percent‚17.368125915527344 network.recv‚463981! disk.dm-0.out‚15398.076171875 disk.dm-1.in‚0$ proc.memory.availableMB‚245.1875 cpu‚26.001030301155335 proc.cpu.threads‚17) disk./.usagePercent‚76.65487617724999% disk./.usageGB‚1176.8439865112305 disk.md0.in‚0 disk.dm-1.out‚2030.880859375-¶•ž ‚(:“ ÀÁôÒó²­Ì gpu.0.gpu‚85.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚268.035' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.34500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T—ùLž ‚(:Á ÏÁôÒÎ…ÜË cpu‚26.024580878349767& disk.nvme1c1n1.out‚4442.8876953125) disk./.usagePercent‚76.65579553109352% disk./.usageGB‚1176.8581008911133 disk.rbd0.in‚0% disk.nvme0n1.out‚13952.8955078125 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 network.sent‚ 4.21726e+06' disk.nvme0c0n1.out‚13952.8955078125 disk.md0.out‚46.53125! disk.dm-0.out‚16265.302734375 disk.dm-1.in‚0$ memory_percent‚98.50215911865234* proc.memory.percent‚17.368125915527344 proc.cpu.threads‚17 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚4442.8876953125 network.recv‚467957 disk.rbd0.out‚0 disk.nvme0n1.in‚ 4.3359375 disk.dm-1.out‚2031.044921875% proc.memory.availableMB‚ 245.40625 proc.memory.rssMB‚ 2845.59375 disk.nvme0c0n1.in‚ 4.3359375"f­žŸ ‚(:“ ÏÁôÒÛû”Ì gpu.0.gpu‚42.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚170.108' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚56.702666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"ÑWÏ2  - _wandbruntime‚1622  _runtime‚1622À;KF¡ ‘{ 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983334541320801 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1167757511138916 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1437130719423294 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.651282012462616 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.32876914739608765 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5610420107841492 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6447502970695496 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 particle/accuracy_4_of_5‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.65495765209198 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0714285746216774 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.28604742884635925 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.35227271914482117 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_2_of_5‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48962751030921936 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 _runtime‚1622.741152217 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_2_of_3‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949056506156921 +particle/sensitivity‚0.6204720735549927 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9962893724441528 particle/accuracy_1_of_5‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4654104709625244 jet/accuracy_5_of_5‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14752799272537231 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.58543]΂+Q532447052002 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5899057984352112 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6226121187210083 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6298075914382935 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 &particle/accuracy‚0.84550541639328 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0714285746216774 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8504472374916077 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5573157072067261 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9983673095703125 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13693898916244507 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.15013515949249268 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.148647278547287 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 trainer/global_step‚3203 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_1_of_2‚0.7480140924453735 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12929461896419525 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 _step‚259 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +particle/specificity‚0.9070436358451843 % _timestamp‚1.7844881461658137e+09 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3928344249725342 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7175437808036804 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11035826057195663 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1628420352935791 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09108603000640869 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15614688396453857 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007619442883878946 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.16696780920028687 +jet/accuracy_3_of_3‚0.06334750354290009 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5388943552970886 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9948981404304504 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5003502368927002 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5597454905509949 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.49842575192451477 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9919690489768982 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5920738577842712 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6013590693473816 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9857583045959473 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5674721598625183 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 /particle/accuracy_3_of_4‚0.9970238208770752 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7670024633407593 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9958670735359192 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9642857313156128 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5410897731781006 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6051945686340332 )loss/val_total_loss‚9.937773612592967 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5948485136032104 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6148535013198853 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +jet/accuracy_2_of_2‚0.12061689794063568 particle/accuracy_2_of_2‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 *jet/accuracy_1_of_4‚0.9505282640457153 particle/accuracy_2_of_4‚1 7validation_average_jet_accuracy‚0.32001346349716187 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.4364526867866516 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8462841510772705 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5212923884391785 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.23595649003982544 'particle/f_score‚0.6329500079154968 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7274649143218994 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13444633781909943 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6762810349464417 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9872881770133972 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6116768717765808 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5586918592453003 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16363798081874847 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9745762944221497 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7589743137359619 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988247811794281 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6867817640304565 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5588615536689758 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9984127283096313 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13690704107284546 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5611575245857239 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988840222358704 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9938238263130188 )jet/accuracy_3_of_4‚0.095587357878685 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7025998830795288 epoch‚3 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21110203862190247 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9983673095703125 *jet/accuracy_1_of_1‚0.2722873091697693 .particle/accuracy_4_of_4‚0.941667914390564 jet/accuracy_3_of_5‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1643928438425064 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3262249529361725 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.34259259700775146 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.27829939126968384 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14824391901493073 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11464047431945801 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09108603000640869 particle/accuracy_1_of_4‚1 jet/accuracy_1_of_5‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9865424633026123 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5785019993782043 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13773253560066223 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1450529247522354 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25198617577552795 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6520451903343201 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.014702044427394867 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 .particle/accuracy_3_of_3‚0.999514102935791 6 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.569444477558136 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9890129566192627 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.995311439037323 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6420632600784302 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6189128756523132 *jet/accuracy_1_of_3‚0.9014416933059692 *jet/accuracy_2_of_4‚0.6223242878913879 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7926691174507141 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7789473533630371 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.56438809633255 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6762810349464417 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8970254063606262 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9953989386558533 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6021984815597534 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9990079998970032 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5358250737190247 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10045591741800308 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7900000214576721 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_3‚1 8"Purity/2FRt0SRqqt1FBt/event_purity‚0.236111119389534 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_5‚1 particle/accuracy_5_of_5‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5465952754020691 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014292909763753414 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.027047201991081238 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15138749778270721 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1353519856929779 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9131747484207153 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5002316236495972 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13444633781909943 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.707626461982727 +jet/accuracy_4_of_4‚0.01071428507566452 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7900000214576721 jet/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.12693022191524506 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.12929193675518036 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.10926788300275803 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14769971370697021 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9160096049308777 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15087711811065674 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.014040268957614899 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9125000238418579 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02408120408654213 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6901792287826538 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5980966687202454 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988389015197754 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9937564134597778 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10000000149011612 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7"Purity/0FRt1SRqqt0FBt/event_purity‚0.56438809633255 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5182291269302368 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48500001430511475 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2076837420463562 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11246827989816666 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172ÏÕxœ=¢ –{ =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7025998830795288 epoch‚3 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21110203862190247 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9983673095703125 *jet/accuracy_1_of_1‚0.2722873091697693 .particle/accuracy_4_of_4‚0.941667914390564 jet/accuracy_3_of_5‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1643928438425064 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3262249529361725 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.34259259700775146 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.27829939126968384 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14824391901493073 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11464047431945801 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09108603000640869 particle/accuracy_1_of_4‚1 jet/accuracy_1_of_5‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9865424633026123 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5785019993782043 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13773253560066223 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1450529247522354 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25198617577552795 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6520451903343201 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.014702044427394867 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 .particle/accuracy_3_of_3‚0.999514102935791 6 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.569444477558136 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9890129566192627 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.995311439037323 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6420632600784302 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6189128756523132 *jet/accuracy_1_of_3‚0.9014416933059692 *jet/accuracy_2_of_4‚0.6223242878913879 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7926691174507141 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7789473533630371 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.56438809633255 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6762810349464417 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8970254063606262 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9953989386558533 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6021984815597534 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9990079998970032 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5358250737190247 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10045591741800308 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7900000214576721 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_3‚1 8"Purity/2FRt0SRqqt1FBt/event_purity‚0.236111119389534 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_5‚1 particle/accuracy_5_of_5‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5465952754020691 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014292909763753414 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.027047201991081238 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15138749778270721 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1353519856929779 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9131747484207153 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5002316236495972 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13444633781909943 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.707626461982727 +jet/accuracy_4_of_4‚0.01071428507566452 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7900000214576721 jet/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.12693022191524506 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.12929193675518036 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.10926788300275803 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14769971370697021 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9160096049308777 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15087711811065674 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.014040268957614899 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9125000238418579 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02408120408654213 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6901792287826538 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5980966687202454 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988389015197754 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9937564134597778 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10000000149011612 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7"Purity/0FRt1SRqqt0FBt/event_purity‚0.56438809633255 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5182291269302368 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48500001430511475 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2076837420463562 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11246827989816666 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983334541320801 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1167757511138916 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1437130719423294 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.651282012462616 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.32876914739608765 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5610420107841492 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6447502970695496 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 particle/accuracy_4_of_5‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.65495765209198 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0714285746216774 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.28604742884635925 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.35227271914482117 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_2_of_5‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48962751030921936 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 _runtime‚1622.741152217 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_2_of_3‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949056506156921 +particle/sensitivity‚0.6204720735549927 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9962893724441528 particle/accuracy_1_of_5‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4654104709625244 jet/accuracy_5_of_5‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14752799272537231 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5854332447052002 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5899057984352112 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6226121187210083 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6298075914382935 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 &particle/accuracy‚0.84550541639328 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0714285746216774 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8504472374916077 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5573157072067261 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9983673095703125 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13693898916244507 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.15013515949249268 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.148647278547287 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 trainer/global_step‚3203 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_1_of_2‚0.7480140924453735 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12929461896419525 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 _step‚259 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +particle/specificity‚0.9070436358451843 % _timestamp‚1.7844881461658137e+09 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3928344249725342 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7175437808036804 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11035826057195663 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1628420352935791 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09108603000640869 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15614688396453857 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007619442883878946 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.16696780920028687 +jet/accuracy_3_of_3‚0.06334750354290009 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5388943552970886 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9948981404304504 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5003502368927002 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5597454905509949 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.49842575192451477 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9919690489768982 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5920738577842712 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6013590693473816 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9857583045959473 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5674721598625183 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 /particle/accuracy_3_of_4‚0.9970238208770752 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7670024633407593 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9958670735359192 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9642857313156128 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5410897731781006 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6051945686340332 )loss/val_total_loss‚9.937773612592967 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5948485136032104 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6148535013198853 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +jet/accuracy_2_of_2‚0.12061689794063568 particle/accuracy_2_of_2‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 *jet/accuracy_1_of_4‚0.9505282640457153 particle/accuracy_2_of_4‚1 7validation_average_jet_accuracy‚0.32001346349716187 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.4364526867866516 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8462841510772705 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5212923884391785 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.23595649003982544 'particle/f_score‚0.6329500079154968 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7274649143218994 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13444633781909943 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6762810349464417 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9872881770133972 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6116768717765808 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5586918592453003 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16363798081874847 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9745762944221497 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7589743137359619 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988247811794281 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6867817640304565 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5588615536689758 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9984127283096313 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13690704107284546 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5611575245857239 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988840222358704 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9938238263130188 )jet/accuracy_3_of_4‚0.095587357878685ƒÕiZôX£ ‚(:Í ÞÁôÒ’ÈŽÌ disk.nvme1c1n1.in‚0 disk.nvme0c0n1.in‚ 4.3359375" proc.memory.rssMB‚ 2844.8828125* proc.memory.percent‚17.363786697387695 cpu‚26.024647438829202' disk.nvme1c1n1.out‚4689.54833984375 disk.nvme1n1.in‚0% disk.nvme1n1.out‚4689.54833984375 disk.md0.out‚ 46.81640625 disk.dm-1.out‚2276.591796875& proc.memory.availableMB‚ 741.328125 proc.cpu.threads‚17 disk.md0.in‚0" disk.dm-0.out‚16319.7197265625( disk.nvme0c0n1.out‚14006.84033203125 network.recv‚471435) disk./.usagePercent‚76.58490937417308% disk./.usageGB‚1175.7698211669922 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚95.47529220581055 network.sent‚ 4.233025e+06 disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚14006.84033203125¼)’¤ ‚(:‡ ÞÁôÒó©ÃÌ gpu.0.gpu‚11.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚54.0 gpu.0.powerWatts‚268.551' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚89.517 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎÙ»Ï2¥ - _wandbruntime‚1645  _runtime‚1645+w©É¦ à ' lr-AdamW/pg2‚0.00021081274413851199 trainer/global_step‚3249 % _timestamp‚1.7844881691443858e+09 _runtime‚1645.719392586 _step‚260 ' lr-AdamW/pg1‚0.00021081274413851199kÙ“"Χ È _step‚260 ' lr-AdamW/pg1‚0.00021081274413851199 ' lr-AdamW/pg2‚0.00021081274413851199 trainer/global_step‚3249 % _timestamp‚1.7844881691443858e+09 _runtime‚1645.719392586„L-÷±2¨ - _wandbruntime‚1645  _runtime‚1645Éï© ÜO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58215472 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3122525387776 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200973046598144 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252053725524480 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32865334 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8610580 VHDeviceStatsMonitor.on_tra¤ð¯ó!in_batch_start/inactive_split_bytes.all.current‚ 161788928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚248931200136704 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32875477 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚249437827609316 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58225617 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 _step‚261 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25350138 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25871113 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚246320932775280 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 137364480 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3117251796718 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197753504732672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17260242 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚246320915735920 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3116911873396 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25870820 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚252054083829248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚252053725524480 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17260242 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚246320915735920 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3116911873396 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25870820 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚252054083829248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚252053725524480 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58215472 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3122525387776 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200973046598144 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252053725524480 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32865334 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8610580 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161788928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚248931200136704 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32875477 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚249437827609316 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58225617 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 _step‚261 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25350138 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25871113 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚246320932775280 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 137364480 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3117251796718 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197753504732672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0…HFÎÍ2« - _wandbruntime‚1646  _runtime‚1646¶Ç•í&¬ çM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚252129205516800 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚249512181915032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58232849 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚249005391905792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚1646.080210951 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3123796092416 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚252128847204864 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25357642 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25357640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3123455299072 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚201033037617664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3118180020066 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 % _timestamp‚1.7844881695050502e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58232849 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58243008 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8613370 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8613079 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚197812381065216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25878400 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚249512538877826 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚249005409424384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3220632128000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25357642 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3220501062144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3123455299072 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25357640 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58232849 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚249005391905792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚1646.080210951 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3123796092416 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚252128847204864 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25357642 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25357640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3123455299072 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚201033037617664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3118180020066 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 % _timestamp‚1.7844881695050502e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58232849 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58243008 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8613370 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8613079 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚197812381065216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25878400 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚249512538877826 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚249005409424384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3220632128000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25357642 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3220501062144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3123455299072 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25357640 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 trainer/global_step‚3299 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚252640788591104 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚200697321359872 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚253155574798324 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25725240 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26251014 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚1670.990498385 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚252640806109696 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3169393995776 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚255810200105472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25725238 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26250731 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8736441 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚249992219859920 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159691776 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3267637318144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59086174 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25725238 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚255809841800704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3169393995776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17514292 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17514290 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3163354938404 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3267502050816 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3163694861726 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚255810200105472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚203964983102464 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33371079 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3169053209600 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33371079±~´rè'º âO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3169393995776 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚255810200105472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25725238 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26250731 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8736441 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚249992219859920 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159691776 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3267637318144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59086174 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25725238 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚255809841800704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3169393995776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17514292 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17514290 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3163354938404 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3267502050816 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3163694861726 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚255810200105472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚203964983102464 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33371079 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3169053209600 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33371079 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 trainer/global_step‚3299 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚252640788591104 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚200697321359872 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚253155574798324 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25725240 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26251014 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚1670.990498385 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚252640806109696 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0‰%xbz2» - _wandbruntime‚1671  _runtime‚1671ÉLaf¼ çM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚255884963481088 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚1671.364965402 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25732742 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚255885321793024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17519373 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204024658690048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚zê8‚‡!0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3164283161640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3170323914240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59113710 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚200756222116864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 % _timestamp‚1.7844881947898386e+09 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25732742 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚250065662981760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚250065645942400 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33370811 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255885321793024 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33380968 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59113710 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59103551 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3268587869184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33380968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26258518 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26258233 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8739145 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚253230286066834 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚253229929104040 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚252714980360192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3169983120896 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720ÿ³ò&½ ìM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚255884963481088 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚1671.364965402 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25732742 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚255885321793024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17519373 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204024658690048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3164283161640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3170323914240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59113710 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚200756222116864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 % _timestamp‚1.7844881947898386e+09 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25732742 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚250065662981760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚250065645942400 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33370811 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255885321793024 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33380968 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59113710 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59103551 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3268587869184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33380968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26258518 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26258233 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8739145 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚253230286066834 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚253229929104040 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚252714980360192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3169983120896 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720Š¹áÉ¥2¾ - _wandbruntime‚1671  _runtime‚1671üÇš ¿ † 0loss/FRt1/assignment_loss‚0.9877473208297223 2loss/SRqqt2/assignment_loss‚0.6417929077130818 /loss/FRt1/detection_loss‚1.3390920018206773 % _timestamp‚1.7844881947909226e+09 _runtime‚1671.365684514 2loss/SRqqt1/assignment_loss‚0.5335167258050775 4loss/FBt2/assignment_loss‚0.00042056551343158877 %loss/total_loss‚9.115295745885195 trainer/global_step‚3299 0loss/FRt2/assignment_loss‚2.3329561002035804 2loss/FBt1/assignment_loss‚0.018537100643137853 1loss/SRqqt1/detection_loss‚0.6042922129235665 1loss/SRqqt2/detection_loss‚0.8903617461992277 epoch‚4 _step‚267 /loss/FRt2/detection_loss‚0.5368880254032363 0loss/FBt1/detection_loss‚0.45123092592862174 /loss/FBt2/detection_loss‚0.7784601129018344˜F×fÀ ‹ % _timestamp‚1.7844881947909226e+09 _runtime‚1671.365684514 2loss/SRqqt1/assignment_loss‚0.5335167258050775 4loss/FBt2/assignment_loss‚0.00042056551343158877 %loss/total_loss‚9.115295745885195 trainer/global_step‚3299 0loss/FRt2/assignment_loss‚2.3329561002035804 2loss/FBt1/assignment_loss‚0.018537100643137853 1loss/SRqqt1/detection_loss‚0.6042922129235665 1loss/SRqqt2/detection_loss‚0.8903617461992277 epoch‚4 _step‚267 /loss/FRt2/detection_loss‚0.5368880254032363 0loss/FBt1/detection_loss‚0.45123092592862174 /loss/FBt2/detection_loss‚0.7784601129018344 0loss/FRt1/assignment_loss‚0.9877473208297223 2loss/SRqqt2/assignment_loss‚0.6417929077130818 /loss/FRt1/detection_loss‚1.3390920018206773‹–y§sLÁ ‚(:Á ‹ÂôÒ²åÛË disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0! disk.dm-0.out‚17113.814453125% disk./.usageGB‚1176.8337440490723 disk.nvme0n1.in‚ 4.3359375# disk.nvme1n1.out‚5513.990234375 disk.md0.out‚47.9375( proc.memory.availableMB‚ 742.01953125" proc.memory.rssMB‚ 2844.8828125 network.sent‚ 4.344207e+06 disk.dm-1.in‚0 disk.rbd0.in‚0% disk.nvme1c1n1.out‚5513.990234375 disk.nvme1n1.in‚0 disk.dm-0.in‚ 4.3359375$ memory_percent‚95.47107219696045 cpu‚26.026072086988858& disk.nvme0c0n1.out‚14759.728515625$ disk.nvme0n1.out‚14759.728515625 disk.md0.in‚0* proc.memory.percent‚17.363786697387695 proc.cpu.threads‚17 network.recv‚483374( disk./.usagePercent‚76.6542090245284 disk.dm-1.out‚3057.412109375óší‘ ‚(:† ‹ÂôÒë»ÉË gpu.0.gpu‚90.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚54.0 gpu.0.powerWatts‚145.95' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚48.65 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t ÈÚXà ‚(:Í šÂôÒª“ÇË cpu‚26.026579420725625% disk./.usageGB‚1176.8373680114746 disk.nvme0n1.in‚ 4.3359375' disk.nvme1c1n1.out‚5515.20361328125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375' proc.memory.availableMB‚ 742.2578125" proc.memory.rssMB‚ 2844.8828125* proc.memory.percent‚17.363786697387695 network.recv‚487734% disk.nvme0n1.out‚15072.8056640625 disk.dm-1.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚76.65444507483957 disk.nvme1c1n1.in‚0 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚17426.72900390625$ memory_percent‚95.46961784362793 proc.cpu.threads‚17 network.sent‚ 4.406735e+06 disk.nvme1n1.in‚0% disk.nvme1n1.out‚5515.20361328125 disk.md0.in‚0 disk.md0.out‚ 48.47265625 disk.dm-1.out‚3057.666015625' disk.nvme0c0n1.out‚15072.8056640625 ©®ŸÄ ‚(:” šÂôÒ¨ßŒÌ gpu.0.gpu‚91.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚124.532' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚41.510666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\ì¦72Å - _wandbruntime‚1695  _runtime‚1695Ö/±±ÉÆ Ã ' lr-AdamW/pg1‚0.00021064355270818423 ' lr-AdamW/pg2‚0.00021064355270818423 trainer/global_step‚3349 % _timestamp‚1.7844882191610243e+09 _runtime‚1695.735989233 _step‚268z•g|ÎÇ È % _timestamp‚1.7844882191610243e+09 _runtime‚1695.735989233 _step‚268 ' lr-AdamW/pg1‚0.00021064355270818423 ' lr-AdamW/pg2‚0.00021064355270818423 trainer/global_step‚3349ŒÞêI2È - _wandbruntime‚1695  _runtime‚1695”F4î%É ÝO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3315529860096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259565958076928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚206956692271616 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚203641137987072 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26100340 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17768340 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚302 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 137364480 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚256873321987332 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26100338 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3215921817600 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26630254 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚259565958076928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3315392495616 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59956876 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚259566316381696 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33856538 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 % _timestamp‚1.7844882191618617e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26100340 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚206956530482688 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33866681 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚256350377045504 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33856538 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8861912 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3209798003412 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚259566316381696 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59967021 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161788928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚253663541023280 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3215921817600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 % _timestamp‚1.7844882191618617e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26100340 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚206956530482688 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33866681 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚256350377045504 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33856538 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8861912 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3209798003412 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚259566316381696 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59967021 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161788928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚253663541023280 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3215921817600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3315529860096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259565958076928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚206956692271616 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚203641137987072 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26100340 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17768340 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚302 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 137364480 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚256873321987332 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26100338 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3215921817600 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26630254 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚259565958076928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3315392495616 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59956876 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚259566316381696 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33856538 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0ÚsÓ52Ë - _wandbruntime‚1696  _runtime‚1696ÜÕ-ì&Ì æM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59974253 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚256424568814592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚259641438069248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33876570 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚297 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3316485938688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141551616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3216510942720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚295 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚256424568814592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3210726226648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8864411 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17773421 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8864116 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚203700038744064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚259641438069248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26107842 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59984412 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26637537 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚253736950066400 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚270 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3349 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3216851736064 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33876570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚256424586333184 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59984412 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59974253 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3316344387072 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33866413 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26637834 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 165976064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚203700014319616 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 _runtime‚1696.407853739Šy]ñ&Í ëM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59984412 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26637537 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚253736950066400 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚270 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3349 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3216851736064 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33876570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚256424586333184 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59984412 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59974253 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3316344387072 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33866413 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26637834 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 165976064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚203700014319616 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 _runtime‚1696.407853739 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59974253 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚256424568814592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚259641438069248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33876570 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚297 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3316485938688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141551616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3216510942720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚295 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚256424568814592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3210726226648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8864411 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17773421 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8864116 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚203700038744064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚259641438069248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26107842 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592ŽFêxê2Î - _wandbruntime‚1696  _runtime‚1696“åûUÏ  epoch‚4 1loss/FBt1/assignment_loss‚0.08157017583466898 1loss/SRqqt1/detection_loss‚0.7678284147383238 % _timestamp‚1.7844882198337119e+09 _step‚271 /loss/FRt2/assignment_loss‚2.307664778803577 2loss/SRqqt2/assignment_loss‚0.6505514211028874 /loss/FBt2/detection_loss‚0.9088593281339854 %loss/total_loss‚9.798025440032118 trainer/global_step‚3349 _runtime‚ 1696.40837919 /loss/FRt1/assignment_loss‚1.092158037194839 2loss/SRqqt1/assignment_loss‚0.5288018569663109 4loss/FBt2/assignment_loss‚0.00017439976190658513 /loss/FRt1/detection_loss‚1.3940842998303615 /loss/FRt2/detection_loss‚0.5694102102541732 1loss/SRqqt2/detection_loss‚1.0002903168299986 /loss/FBt1/detection_loss‚0.4966322005810861û­¸ž Ð † epoch‚4 1loss/FBt1/assignment_loss‚0.08157017583466898 1loss/SRqqt1/detection_loss‚0.7678284147383238 % _timestamp‚1.7844882198337119e+09 _step‚271 /loss/FRt2/assignment_loss‚2.307664778803577 2loss/SRqqt2/assignment_loss‚0.6505514211028874 /loss/FBt2/detection_loss‚0.9088593281339854 %loss/total_loss‚9.798025440032118 trainer/global_step‚3349 _runtime‚ 1696.40837919 /loss/FRt1/assignment_loss‚1.092158037194839 2loss/SRqqt1/assignment_loss‚0.5288018569663109 4loss/FBt2/assignment_loss‚0.00017439976190658513 /loss/FRt1/detection_loss‚1.3940842998303615 /loss/FRt2/detection_loss‚0.5694102102541732 1loss/SRqqt2/detection_loss‚1.0002903168299986 /loss/FBt1/detection_loss‚0.4966322005810861¸Q‹k©Ñ ‚(:É ©ÂôÒÌüË# disk.dm-0.out‚17430.48291015625 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0 cpu‚26.024030513340325 disk.md0.in‚0 disk.dm-1.iŽ5S«n‚0 disk.nvme0c0n1.in‚ 4.3359375# disk.nvme0n1.out‚15076.91796875 disk.nvme1n1.in‚0$ memory_percent‚95.46980857849121" proc.memory.rssMB‚ 2844.8828125 network.recv‚490979 disk.md0.out‚ 48.88671875 disk.dm-0.in‚ 4.3359375 disk.dm-1.out‚3057.833984375 disk.rbd0.out‚0' disk.nvme1c1n1.out‚5515.90185546875' proc.memory.availableMB‚ 742.2265625% disk./.usageGB‚1176.8373374938965% disk.nvme0c0n1.out‚15076.91796875 disk.nvme0n1.in‚ 4.3359375% disk.nvme1n1.out‚5515.90185546875* proc.memory.percent‚17.363786697387695 proc.cpu.threads‚17 network.sent‚ 4.412686e+06) disk./.usagePercent‚76.65444308704748A‘4“Ò ‚(:ˆ ©ÂôÒÙ¸¢Ì gpu.0.gpu‚89.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚142.239' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚47.413 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€"j2Ó - _wandbruntime‚1719  _runtime‚1719ÀßÏ;ÆÔ À $ _timestamp‚1.784488243411896e+09 _runtime‚1719.986955316 _step‚272 & lr-AdamW/pg1‚0.0002105564523924531 & lr-AdamW/pg2‚0.0002105564523924531 trainer/global_step‚3399•µSËÕ Å & lr-AdamW/pg1‚0.0002105564523924531 & lr-AdamW/pg2‚0.0002105564523924531 trainer/global_step‚3399 $ _timestamp‚1.784488243411896e+09 _runtime‚1719.986955316 _step‚272»ÁÄ2Ö - _wandbruntime‚1719  _runtime‚1719Ýðí.â'× ÜO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚260059983018496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚257334845147280 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34362283 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60827578 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60837723 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚209948289518080 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26475440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18022390 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚260591069176340 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27010327 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27010036 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209948447112704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3262449639424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚206584979038720 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3256580991742 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34352140 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚260059965499904 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3256241068420 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚260059983018496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚257334828107920 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8987935 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8987646 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26475438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34352140 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚263322074353152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3262449639424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 $ _timestamp‚1.784488243412749e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26475440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18022392 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3363334903808 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26475438 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚260059965499904 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3262108853248 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8987935 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8987646 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26475438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34352140 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚263322074353152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3262449639424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 $ _timestamp‚1.784488243412749e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26475440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18022392 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3363334903808 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26475438 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚260059965499904 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3262108853248 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚260059983018496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚257334845147280 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34362283 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60827578 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60837723 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚209948289518080 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26475440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18022390 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚260591069176340 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27010327 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27010036 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209948447112704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720‘‰“®2Ù - _wandbruntime‚1720  _runtime‚1720™¾nì&Ú æM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚257408254190400 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚3399 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18027471 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3364421795328 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60844955 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60844955 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34362015 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚260134174787584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3257509215090 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60855114 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34372172 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3257169291656 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34372172 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚257408271229760 $ _timestamp‚1.784488243789014e+09 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚263397554345472 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚260665423482056 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3263379557888 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159684608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚263397554345472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚260134174787584 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26482942 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27017849 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚260134157268992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚206643855371264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18027473 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26482940 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27017559 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794ò:@™Û ëM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60855114 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3263038764544 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 UIDeviceStatsMonitor.on_train_ba™Ê2ÇX$tch_end/inactive_split.small_pool.allocated‚8990376 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 _runtime‚1720.364150815 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3263038764544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3257509215090 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60855114 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34372172 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3257169291656 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34372172 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚257408271229760 $ _timestamp‚1.784488243789014e+09 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚263397554345472 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚260665423482056 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3263379557888 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159684608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚263397554345472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚260134174787584 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26482942 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27017849 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚260134157268992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚206643855371264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18027473 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26482940 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27017559 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚257408254190400 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚3399 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18027471 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3364421795328 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60844955 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60844955 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34362015 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚260134174787584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18276442 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34857885 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18276440 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26850540 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚263769553954304 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61708425 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26850538 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27389992 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 _step‚277 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267078190629376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚263769571472896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3308977461248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3302684133428 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26850538 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34847742 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚209528795665920 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚209528771241472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚1745.618488707 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚267078190629376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3411388000256 trainer/global_step‚3449 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34847742 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3308636675072 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26850540 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚261006149271280 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3308977461248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚264309173328030 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚261006132231920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3411261121536 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 $ _timestamp‚1.784488269039735e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚267078548934144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚212940183666176 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267078548934144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3308636675072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9113832 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9113552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚212940032363008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3411388000256 trainer/global_step‚3449 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34847742 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3308636675072 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26850540 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚261006149271280 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3308977461248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚264309173328030 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚261006132231920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3411261121536 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 $ _timestamp‚1.784488269039735e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚267078548934144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚212940183666176 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267078548934144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3308636675072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9113832 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9113552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚212940032363008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18276442 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34857885 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 TGDeviceSt–ò=µ§atsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18276440 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26850540 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚263769553954304 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61708425 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26850538 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27389992 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 _step‚277 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267078190629376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚263769571472896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3308977461248•Ÿw)'2é - _wandbruntime‚1745  _runtime‚1745|Cì&ê æM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61725816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18281521 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚267153670621696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34857617 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚278 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚209587647574016 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚263843745723392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27397877 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚261079575353760 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61715657 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18281523 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9116354 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3309566586368 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚264383527633858 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26858042 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚267153312309760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26858040 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚263843745723392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61715657 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚264383170671064 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34857617 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 $ _timestamp‚1.784488269412388e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 116385792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3412338072064 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34867774 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚212999869260288 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚209587671998464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27397600 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 trainer/global_step‚3449 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267153670621696 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3309907379712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259£Ç “ñ&ë ëM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61715657 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚264383170671064 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34857617 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 $ _timestamp‚1.784488269412388e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 116385792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3412338072064 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34867774 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚212999869260288 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚209587671998464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27397600 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 trainer/global_step‚3449 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267153670621696 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3309907379712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61725816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18281521 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚267153670621696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34857617 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚278 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚209587647574016 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚263843745723392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27397877 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚261079575353760 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61715657 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18281523 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9116354 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3309566586368 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚264383527633858 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26858042 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚267153312309760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26858040 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚263843745723392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0–î‚ø2ì - _wandbruntime‚1745  _runtime‚1745]lí € /loss/FRt1/assignment_loss‚0.960842980955639 2loss/SRqqt2/assignment_loss‚0.6414236594448048 2loss/FBt1/assignment_loss‚0.009026317740229217 /loss/FRt2/detection_loss‚0.5724195033953117 /loss/FBt1/detection_loss‚0.5067119231694635 %loss/total_loss‚9.267785492074557 % _timestamp‚1.7844882694134328e+09 _runtime‚1745.988101677 0loss/FRt2/assignment_loss‚2.3909302476611614 /loss/FRt1/detection_loss‚1.3472696617177695 0loss/SRqqt1/detection_loss‚0.619392458664356 1loss/SRqqt2/detection_loss‚1.0018813167911729 trainer/global_step‚3449 1loss/SRqqt1/assignment_loss‚0.538790333204966 /loss/FBt2/detection_loss‚0.6781534664332867 epoch‚4 _step‚279 2loss/FBt2/assignment_loss‚0.000943622896396418[#‚ î … %loss/total_loss‚9.267785492074557 % _timestamp‚1.7844882694134328e+09 _runtime‚1745.988101677 0loss/FRt2/assignment_loss‚2.3909302476611614 /loss/FRt1/detection_loss‚1.3472696617177695 0loss/SRqqt1/detection_loss‚0.619392458664356 1loss/SRqqt2/detection_loss‚1.0018813167911729 trainer/global_step‚3449 1loss/SRqqt1/assignment_loss‚0.538790333204966 /loss/FBt2/detection_loss‚0.6781534664332867 epoch‚4 _step‚279 2loss/FBt2/assignment_loss‚0.000943622896396418 /loss/FRt1/assignment_loss‚0.960842980955639 2loss/SRqqt2/assignment_loss‚0.6414236594448048 2loss/FBt1/assignment_loss‚0.009026317740229217 /loss/FRt2/detection_loss‚0.5724195033953117 /loss/FBt1/detection_loss‚0.5067119231694635—|,EŒï ‚(:’ ÖÂôÒ‡é•Ì gpu.0.gpu‚90.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚244.49' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚81.49666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î†]Rð ‚(:Ç ÖÂôÒ¥»âÌ network.sent‚ 4.55361e+06 network.recv‚502416' disk./.usagePercent‚76.654496260486& disk.nvme0n1.out‚15086.97998046875# proc.memory.availableMB‚741.875% disk./.usageGB‚1176.8381538391113% disk.nvme1n1.out‚5520.05810546875 disk.md0.out‚ 49.9296875! disk.dm-1.out‚3059.9931640625$ memory_percent‚95.47195434570312 cpu‚26.029456340988556 disk.nvme0c0n1.in‚ 4.3359375( disk.nvme0c0n1.out‚15086.97998046875 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0 disk.dm-0.in‚ 4.3359375 disk.rbd0.in‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2844.8828125' disk.nvme1c1n1.out‚5520.05810546875" disk.dm-0.out‚17440.2978515625 disk.dm-1.in‚0* proc.memory.percent‚17.363786697387695 proc.cpu.threads‚17 disk.nvme1n1.in‚0 disk.md0.in‚0ɇ=*Rñ ‚(:Ç åÂôÒ»òË" proc.memory.rssMB‚ 2844.8828125 disk.rbd0.out‚0& disk.nvme0n1.out‚15087.82666015625" disk.dm-0.out‚17440.8994140625# memory_percent‚95.4704999923706% disk./.usageGB‚1176.5999374389648( disk.nvme0c0n1.out‚15087.82666015625 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚5520.59912109375% disk.nvme1n1.out‚5520.59912109375 disk.md0.out‚ 50.22265625 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375( proc.memory.availableMB‚ 742.11328125* proc.memory.percent‚17.363786697387695 cpu‚26.035401763041186 network.sent‚ 4.604145e+06 network.recv‚505962( disk./.usagePercent‚76.6389798038733 disk.rbd0.in‚0 disk.nvme0n1.in‚ 4.3359375 disk.dm-1.out‚3060.125³$ÒÇò ‚(:’ åÂôÒé«Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚118.202' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.400666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+kcÐ2ó - _wandbruntime‚1770  _runtime‚17705õóÞÉô à ' lr-AdamW/pg1‚0.00021037724953527575 ' lr-AdamW/pg2‚0.00021037724953527575 trainer/global_step‚3499 % _timestamp‚1.7844882937136617e+09 _runtime‚1770.289170286 _step‚280‚ÙøÎõ È _runtime‚1770.289170286 _step‚280 ' lr-AdamW/pg1‚0.00021037724953527575 ' lr-AdamW/pg2‚0.00021037724953527575 trainer/global_step‚3499 % _timestamp‚1.7844882937136617e+09˜·òÈ2ö - _wandbruntime‚1770  _runtime‚1770MxËó¾÷ ÝO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3355505283072 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3355505283072 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚212472587868672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18530492 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚267479159927296 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27225640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3355164496896 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3459342932480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35353487 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚215931799728128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚268026920517038 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9240757 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚267479159927296 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27225638 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3459211859456 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35353487 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚212472612293120 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚264677453395280 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚270834306905600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚264677436355920 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 trainer/global_step‚3499 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚267479142408704 TDDeviceStatsMonitor.o?¯‰2%n_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27771249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3355164496896 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7844882937143211e+09 _step‚281 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497472 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 _runtime‚1770.289417576 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62568982 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27225638 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚270834665210368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35343344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚270834665210368 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35343344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27770960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18530490 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚268026563554356 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3355505283072 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3355505283072 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚212472587868672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18530492 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚267479159927296 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27225640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3355164496896 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3459342932480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35353487 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚215931799728128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚268026920517038 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9240757 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚267479159927296 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27225638 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3459211859456 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35353487 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚212472612293120 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚264677453395280 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚270834306905600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚264677436355920 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 trainer/global_step‚3499 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚267479142408704 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27771249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3355164496896 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7844882937143211e+09 _step‚281 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497472 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 _runtime‚1770.289417576 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62568982 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27225638 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚270834665210368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35343344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚270834665210368 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35343344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27770960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18530490 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚268026563554356 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚264750879477760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3356094408192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9243326 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 % _timestamp‚1.7844882940729961e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27233142 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3356094408192 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚215991792041984 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3460172350464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18535573 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚264750862438400 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62596518 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚268101274822866 _step‚282 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3356435201536 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62586359 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9243033 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚212531488625664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35353219 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27778604 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚212531464201216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27233140 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚267553351696384 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27778899 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0†‡6±!û ìM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3460303416320 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27778604 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚212531464201216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27233140 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚267553351696384 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27778899 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚268100917860072 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62586359 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚267553334177792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚264750879477760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3356094408192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9243326 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 % _timestamp‚1.7844882940729961e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27233142 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3356094408192 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚215991792041984 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3460172350464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18535573 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚264750862438400 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62596518 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚268101274822866 _step‚282 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3356435201536 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62586359 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9243033 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚212531488625664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35353219 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35838946 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3401692318720 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚218923572669952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚271188748381696 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18784540 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27600738 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3402033104896 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3395910186766 _step‚285 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚274590781486592 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9367442 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63449829 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27600740 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3395570263444 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 _runtime‚1794.746528646 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚218923721875968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚271188748381696 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3402033104896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28152276 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124781568 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚274590423181824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18784542 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63439684 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚274590781486592 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28151982 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚268348757519280 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35849089 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3507168174080 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚271744667706046 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚93677344Û¦'è'† âO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚218923721875968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚271188748381696 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3402033104896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28152276 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124781568 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚274590423181824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18784542 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63439684 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚274590781486592 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28151982 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚268348757519280 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35849089 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3507168174080 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚271744667706046 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9367734 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35838946 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3401692318720 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚218923572669952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚271188748381696 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18784540 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27600738 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3402033104896 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3395910186766 _step‚285 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚274590781486592 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9367442 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63449829 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27600740 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3395570263444 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 _runtime‚1794.746528646'-PÖ2‡ - _wandbruntime‚1795  _runtime‚1795 ~JI/ˆ çM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚274665903174144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚218983557336576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3402622230016 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚271262940150784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27608240 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚271262940150784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35848821 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚271262922632192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3396498486680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3508123115008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚271262922632192 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚215475305252864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 _step‚286 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27608240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18789621 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚215475280828416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3396838410114 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚299 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 % _timestamp‚1.7844883185300035e+09 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚268422166562400 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27608242 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚274665544862208 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚271818665049080 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63467220 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3402622230016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27608242 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚268422183601760 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28159583 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3402963023360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 _runtime‚1795.105162898 VIDeviceStatsMonitor.on_train_batcGs©ó¾ h_end/inactive_split.large_pool.allocated‚18789623 trainer/global_step‚3549 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63457061 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35858978 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9369962 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚218983403943424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63467220 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚274665903174144 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3402963023360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9370259 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3508252083712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63457061 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368¢ŠTãò&‰ ìM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18789621 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚215475280828416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3396838410114 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚299 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 % _timestamp‚1.7844883185300035e+09 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚268422166562400 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27608242 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚274665544862208 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚271818665049080 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63467220 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3402622230016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27608242 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚268422183601760 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28159583 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3402963023360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 _runtime‚1795.105162898 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18789623 trainer/global_step‚3549 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63457061 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35858978 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9369962 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚218983403943424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63467220 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚274665903174144 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3402963023360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9370259 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3508252083712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63457061 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚274665903174144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚218983557336576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3402622230016 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚271262940150784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27608240 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚271262940150784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35848821 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚271262922632192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3396498486680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3508123115008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚271262922632192 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚215475305252864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 _step‚286 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27608240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784ž¥Ù¨2Š - _wandbruntime‚1795  _runtime‚1795/×댋 ÿ $ _timestamp‚1.784488318530954e+09 2loss/SRqqt2/assignment_loss‚0.6126824218080225 /loss/FRt1/detection_loss‚1.3723955110913937 1loss/SRqqt1/detection_loss‚0.6592298430352195 0loss/SRqqt2/detection_loss‚1.147192281531865 /loss/FBt2/detection_loss‚0.7812513077010711 _step‚287 0loss/FRt1/assignment_loss‚0.9661442164880821 /loss/FBt1/detection_loss‚0.7721916606662885 trainer/global_step‚3549 _runtime‚1795.105692468 1loss/FBt2/assignment_loss‚0.10430025074095586 /loss/FRt2/detection_loss‚0.5889772650384111 /loss/FRt2/assignment_loss‚2.218114778637886 2loss/SRqqt1/assignment_loss‚0.6134337733469143 2loss/FBt1/assignment_loss‚0.016598332795124248 %loss/total_loss‚9.852511642881232 epoch‚4Eªë‰ Œ „ /loss/FRt2/assignment_loss‚2.218114778637886 2loss/SRqqt1/assignment_loss‚0.6134337733469143 2loss/FBt1/assignment_loss‚0.016598332795124248 %loss/total_loss‚9.852511642881232 epoch‚4 $ _timestamp‚1.784488318530954e+09 2loss/SRqqt2/assignment_loss‚0.6126824218080225 /loss/FRt1/detection_loss‚1.3723955110913937 1loss/SRqqt1/detection_loss‚0.6592298430352195 0loss/SRqqt2/detection_loss‚1.147192281531865 /loss/FBt2/detection_loss‚0.7812513077010711 _step‚287 0loss/FRt1/assignment_loss‚0.9661442164880821 /loss/FBt1/detection_loss‚0.7721916606662885 trainer/global_step‚3549 _runtime‚1795.105692468 1loss/FBt2/assignment_loss‚0.10430025074095586 /loss/FRt2/detection_loss‚0.5889772650384111ŸE;ãY ‚(:Î ƒÃôÒŽ±‚Ì disk.nvme1c1n1.in‚0 disk.md0.in‚0$ memory_percent‚95.47116756439209" proc.memory.rssMB‚ 2844.8828125* proc.memory.percent‚17.363786697387695& disk.nvme0n1.out‚15092.59619140625 network.sent‚ 4.682267e+06 network.recv‚513770$ disk./.usageGB‚1172.855712890625 disk.dm-0.in‚ 4.3359375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.3359375$ disk.nvme1n1.out‚5536.6123046875" disk.dm-0.out‚17459.2568359375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚15092.59619140625 disk.md0.out‚ 50.82421875( proc.memory.availableMB‚ 742.00390625 cpu‚26.046113962852903 proc.cpu.threads‚17" disk.dm-1.out‚3061.24169921875 disk.nvme0n1.in‚ 4.3359375& disk.nvme1c1n1.out‚5536.6123046875 disk.nvme1n1.in‚0) disk./.usagePercent‚76.39509610100147 disk.dm-1.in‚0²¯šêžŽ ‚(:“ ƒÃôÒ¤Š´Ì gpu.0.gpu‚94.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚54.0 gpu.0.powerWatts‚256.313' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.43766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôõÈzž ‚(:“ ’ÃôÒë–öË gpu.0.gpu‚68.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚253.762' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.58733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Áv}±X ‚(:Í ’ÃôÒ—àƒÌ* proc.memory.percent‚17.363786697387695 network.recv‚517817 disk.md0.out‚ 51.09765625" disk.dm-1.out‚3061.37060546875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375( proc.memory.availableMB‚ 741.75390625" proc.memory.rssMB‚ 2844.8828125( disk./.usagePercent‚76.3951117548642% disk./.usageGB‚1172.8559532165527$ disk.nvme1n1.out‚5538.2744140625 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚5538.2744140625$ memory_percent‚95.47269344329834 cpu‚26.043821762449074 disk.nvme1n1.in‚0 disk.md0.in‚0# disk.dm-0.out‚17468.28857421875 proc.cpu.threads‚17 network.sent‚ 4.733552e+06 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0' disk.nvme0c0n1.out‚15100.6923828125 disk.nvme0n1.in‚ 4.3359375% disk.nvme0n1.out‚15100.6923828125¡¹Ð)2‘ - _wandbruntime‚1818  _runtime‚1818›…xÉ’ à ' lr-AdamW/pg1‚0.00021019138665593297 ' lr-AdamW/pg2‚0.00021019138665593297 trainer/global_step‚3599 % _timestamp‚1.7844883423852415e+09 _runtime‚1818.960214219 _step‚288úó)AΓ È ' lr-AdamW/pg2‚0.00021019138665593297 trainer/global_step‚3599 % _timestamp‚1.7844883423852415e+09 _runtime‚1818.960214219 _step‚288 ' lr-AdamW/pg1‚0.00021019138665593297 = {ö2” - _wandbruntime‚1818  _runtime‚1818øì2ã'• ÝO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36334548 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚274898319317504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3555077566976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚275462414895054 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28533871 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚278346539458048 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3448220140544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27975838 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚274898336836096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚274898319317504 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚278346897762816 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3448560926720 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36344691 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚218360221123072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3442013328452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9494989 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 _step‚289 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19038590 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27975840 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚221915452090368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64310386 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚221915298690048 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚272020061643280 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3448220140544 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9495279 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚218360245547520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3442353251774 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3448560926720 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 % _timestamp‚1.7844883423861945e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚274898336836096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19038592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 trainer/global_step‚3599 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3555206542848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚278346539458048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚278346539458048 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3448220140544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27975838 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚274898336836096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚274898319317504 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚278346897762816 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3448560926720 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36344691 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚218360221123072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3442013328452 GADeviceStatsMonitor.on_trahñ„n in_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9494989 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 _step‚289 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19038590 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27975840 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚221915452090368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64310386 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚221915298690048 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚272020061643280 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3448220140544 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9495279 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚218360245547520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3442353251774 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3448560926720 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 % _timestamp‚1.7844883423861945e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚274898336836096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19038592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 trainer/global_step‚3599 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3555206542848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚278346539458048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3449490845184 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3442941551688 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚221975289872384 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28541428 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27983340 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3449150051840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚221975134382080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚278422019450368 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚274972511086592 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9497466 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚218419121880064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3449150051840 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9497755 _runtime‚1819.326652747 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3443281475122 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 _step‚290 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚278421661138432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚274972528605184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚272093470686400 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 % _timestamp‚1.7844883427508512e+09 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚272093487725760 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64337922 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚275536412238088 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27983340 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3449490845184 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36354580 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27983342 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36344423 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚274972528605184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274972511086592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28541137 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19043673 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64327763 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 trainer/global_step‚3599 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19043671 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3556036926464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36354580 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259¹üñò&™ ìM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚272093470686400 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 % _timestamp‚1.7844883427508512e+09 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚272093487725760 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64337922 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚275536412238088 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27983340 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3449490845184 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36354580 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27983342 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36344423 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚274972528605184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274972511086592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28541137 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19043673 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64327763 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 trainer/global_step‚3599 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19043671 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3556036926464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36354580 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3449490845184 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3442941551688 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚221975289872384 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28541428 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27983340 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3449150051840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚221975134382080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚278422019450368 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚274972511086592 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9497466 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚218419121880064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3449150051840 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9497755 _runtime‚1819.326652747 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3443281475122 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 _step‚290 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚278421661138432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚274972528605184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448¢¨ÎÐ2š - _wandbruntime‚1819  _runtime‚1819bøàe › ƒ 1loss/SRqqt1/detection_loss‚0.6140801531660423 /loss/FBt2/detection_loss‚1.0593927373679786 0loss/FRt1/assignment_loss‚1.0067139829735499 2loss/FBt2/assignment_loss‚0.013430325354420347 /loss/FRt1/detection_loss‚1.2747483134452202 1loss/SRqqt2/detection_loss‚1.0762238290054458 _runtime‚ 1819.32783388 _step‚291 2loss/SRqqt1/assignment_loss‚0.5895262701768522 3loss/FBt1/assignment_loss‚0.0050723942363859464 &loss/total_loss‚10.163271901597803 trainer/global_step‚3599 % _timestamp‚1.7844883427530143e+09 /loss/FBt1/detection_loss‚0.9570206549643235 epoch‚4 /loss/FRt2/assignment_loss‚2.272569624244744 2loss/SRqqt2/assignment_loss‚0.6976511936009462 /loss/FRt2/detection_loss‚0.5968424230618923ý™œ ˆ &loss/total_loss‚10.163271901597803 trainer/global_step‚3599 % _timestamp‚1.7844883427530143e+09 /loss/FBt1/detection_loss‚0.9570206549643235 epoch‚4 /loss/FRt2/assignment_loss‚2.272569624244744 2loss/SRqqt2/assignment_loss‚0.6976511936009462 /loss/FRt2/detection_loss‚0.5968424230618923 1loss/SRqqt1/detection_loss‚0.6140801531660423 /loss/FBt2/detection_loss‚1.0593927373679786 0loss/FRt1/assignment_loss‚1.0067139829735499 2loss/FBt2/assignment_loss‚0.013430325354420347 /loss/FRt1/detection_loss‚1.2747483134452202 1loss/SRqqt2/detection_loss‚1.0762238290054458 _runtime‚ 1819.32783388 _step‚291 2loss/SRqqt1/assignment_loss‚0.5895262701768522 3loss/FBt1/assignment_loss‚0.0050723942363859464£eç“T ‚(:É ¡ÃôÒ±ìÀË network.sent‚ 4.736474e+06 network.recv‚521159) disk./.usagePercent‚76.39512964499305 disk.rbd0.out‚0 disk.md0.out‚ 51.5078125$ memory_percent‚95.47135829925537" proc.memory.rssMB‚ 2844.8828125* proc.memory.percent‚17.363786697387695 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚5539.005859375# disk.nvme1n1.out‚5539.005859375 disk.md0.in‚0" disk.dm-1.out‚3061.58935546875% disk./.usageGB‚1172.8562278747559& disk.nvme0c0n1.out‚15101.677734375 disk.nvme0n1.in‚ 4.3359375 disk.dm-0.in‚ 4.3359375 cpu‚26.041447064644366 proc.cpu.threads‚17 disk.rbd0.in‚0$ disk.nvme0n1.out‚15101.677734375 disk.nvme1n1.in‚0# disk.dm-0.out‚17468.89404296875 disk.dm-1.in‚0( proc.memory.availableMB‚ 741.97265625*BW‘ž ‚(:† ¡ÃôÒ‘ç¥Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚118.803' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.601 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç ñH2Ÿ - _wandbruntime‚1843  _runtime‚1843¼„¤Ç  Á % _timestamp‚1.7844883669095826e+09 _runtime‚1843.484552935 _step‚292 & lr-AdamW/pg1‚0.0002100959614125017 & lr-AdamW/pg2‚0.0002100959614125017 trainer/global_step‚3649 a"Ì¡ Æ trainer/global_step‚3649 % _timestamp‚1.7844883669095826e+09 _runtime‚1843.484552935 _step‚292 & lr-AdamW/pg1‚0.0002100959614125017 & lr-AdamW/pg2‚0.0002100959614125017¤ƒ D2¢ - _wandbruntime‚1843  _runtime‚1843`‰] [£ ÝO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19292640 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9622820 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36840293 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚279179805121380 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚278607907771904 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚282103014039040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚221304062174720 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚278607907771904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3488796316782 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3488456393460 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚278607925290496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3495088748544 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚282103014039040 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚282102655734272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3603081085440 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65181088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9622531 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚1843.485609757 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3494747962368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.pe¬­ã¦ˆak‚834 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28350940 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚224907118835712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚221304037750272 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28915462 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28350938 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36830150 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36830150 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚224907272236032 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚3649 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28350940 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3494747962368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19292642 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚279180162084062 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚275691348727920 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 _step‚293 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36840293 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚65191233 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚278607925290496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28350938 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7844883669105992e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3495088748544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28915171 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3603210061312 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚282102655734272 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36840293 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚279179805121380 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚278607907771904 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚282103014039040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚221304062174720 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚278607907771904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3488796316782 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3488456393460 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚278607925290496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3495088748544 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚282103014039040 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚282102655734272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3603081085440 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65181088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9622531 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚1843.485609757 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3494747962368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28350940 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚224907118835712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚221304037750272 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28915462 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28350938 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36830150 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36830150 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚224907272236032 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚3649 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28350940 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3494747962368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19292642 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚279180162084062 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚275691348727920 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 _step‚293 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36840293 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚65191233 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚278607925290496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28350938 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7844883669105992e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3495088748544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934¥x\f^2¥ - _wandbruntime‚1843  _runtime‚1843eE Oí&¦ çM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3496018667008 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65208624 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28922788 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚279254159427096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 _step‚294 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚282177777414656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28358440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 _runtime‚1843.854382131 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36850182 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚278682117059584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚278682117059584 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9625357 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133163008 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚65208624 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚282178135726592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3495677873664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28358442 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36840025 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚275764791849760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚65198465 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28923080 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7844883672792196e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3604166797312 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚279254516389890 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚275764774810400 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65198465 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚282178135726592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28358442 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3489384616696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚278682099540992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19297721 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3489724540130 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚278682099540992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19297723 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚221362914082816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3604033634304 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3495677873664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9625067 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088n¡PN§ ìM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚224966947717120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚282177777414656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28358440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 _runtime‚1843.854382131 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36850182 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚278682117059584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚278682117059584 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9625357 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133163008 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚65208624 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚282178135726592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3495677873664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28358442 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36840025 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚275764791849760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚65198465 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28923080 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7844883672792196e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3604166797312 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚279254516389890 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚275764774810400 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65198465 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚282178135726592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28358442 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3489384616696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_ba”xL¤ tch_end/segment.large_pool.peak‚675 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚278682099540992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19297721 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3489724540130 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚278682099540992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19297723 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚221362914082816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3604033634304 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3495677873664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9625067 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3496018667008 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65208624 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28922788 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚279254159427096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 _step‚294 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚285859130315264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚224247878801920 trainer/global_step‚3699 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚297 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9750013 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚227899008583680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚279362652851920 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 _step‚297 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28726038 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚224247854377472 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19546690 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 _runtime‚ 1868.88422378 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3541616570368 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚282317513744896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚227898850989056 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28726040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚282897909273070 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3535239381790 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚285858772010496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37325752 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66051790 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3541616570368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19546692 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚282897552310388 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7844883923090994e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3541275784192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37325752 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28726040 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37335895 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚282317513744896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28726038 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29296406 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚282317496226304 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚282317496226304 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29296705 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9749716 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3650996611584 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37335895 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚285858772010496 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3534899458468 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚285859130315264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚224247878801920 trainer/global_step‚3699 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚297 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9750013 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚227899008583680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚279362652851920 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 _step‚297 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28726038 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚224247854377472 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19546690 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 _runtime‚ 1868.88422378 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3541616570368 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚282317513744896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚227898850989056 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28726040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚282897909273070 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3535239381790 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚285858772010496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37325752 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66051790 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3541616570368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19546692 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚282897552310388 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7844883923090994e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3541275784192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37325752 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28726040 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37335895 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚282317513744896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28726038 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29296406 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚282317496226304 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚282317496226304 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29296705 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9749716 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3650996611584 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37335895 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚285858772010496 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3534899458468 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66079326 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28733542 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 trainer/global_step‚3699 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚285933893690880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28733540 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3542546488832 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚224306755134464 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3652089327616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29303987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19551773 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28733542 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚282391687995392 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3542546488832 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚224306730710016 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9752505 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚282391705513984 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3542205695488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37345784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚282391687995392 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 _runtime‚1869.259207597 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9752216 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37335627 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚279436095973760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚285934252002816 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66069167 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282972263578898 % _timestamp‚1.7844883926840224e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37345784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚279436078934400 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚227958844462080 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚282971906616104 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3542205695488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28733540 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3651958261760 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚282391705513984 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚285934252002816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19551771 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720xÖš;ò&· ìM UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 _step‚298 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚224306730710016 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9752505 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚282391705513984 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3542205695488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37345784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚282391687995392 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 _runtime‚1869.259207597 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9752216 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37335627 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚279436095973760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚285934252002816 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66069167 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282972263578898 % _timestamp‚1.7844883926840224e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37345784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚279436078934400 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚227958844462080 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚282971906616104 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3542205695488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28733540 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3651958261760 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚282391705513984 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚285934252002816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19551771 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3535827681704 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66079326 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28733542 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 trainer/global_step‚3699 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚285933893690880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28733540 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3542546488832 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚224306755134464 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3652089327616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29303987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19551773 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28733542 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚282391687995392 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3542546488832 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66932637 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚286615299499396 trainer/global_step‚3749 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9876151 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3581342523476 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29677186 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚227191695429120 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29101140 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37821354 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚230890651589120 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3588144392192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3581682446798 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚286027102199296 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29676891 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7844884178410783e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚286027084680704 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚286027084680704 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37831497 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29101138 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚286615656462078 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19800740 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37831497 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3587803606016 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚283033974015280 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚1894.416118683 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚289614888286720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66922492 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9876444 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289614888286720 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259°UE¿ªÄ âO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚286027084680704 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37831497 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29101138 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚286615656462078 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19800740 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37831497 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3587803606016 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚283033974015280 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚1894.416118683 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚289614888286720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 IBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚289615246591488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29101140 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3587803606016 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37821354 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3588144392192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3698980584448 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66922492 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9876444 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289614888286720 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚283033956975920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66932637 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚286615299499396 trainer/global_step‚3749 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9876151 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3581342523476 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29677186 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚227191695429120 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29101140 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37821354 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚230890651589120 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3588144392192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3581682446798 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚286027102199296 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29676891 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7844884178410783e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚286027084680704­ÚZ[Ù2Å - _wandbruntime‚1894  _runtime‚1894JõÑì&Æ æM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66939869 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9879023 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3700070961664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3588733517312 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚286101293968384 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3589074310656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29108642 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66950028 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3582610670146 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚286690010767906 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚283107400097760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37831229 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3582270746712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19805821 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚230950642723328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚230950485135872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚227250571761664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚300 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66950028 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3589074310656 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37841386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚289690009967104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66939869 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚289690368279040 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚286101293968384 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 trainer/global_step‚3749 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚286101276449792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚289690368279040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3699937798656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29684546 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19805823 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚298 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29108640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3588733517312 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37831229 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 _runtime‚1894.791574385 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 $ _timestamp‚1.784488418216243e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29108642 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚286689653805112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚227250547337216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133163008 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37841386î= +ñ&Ç ëM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚227250547337216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133163008 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37841386 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66939869 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9879023 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3700070961664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3588733517312 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚286101293968384 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3589074310656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29108642 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66950028 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3582610670146 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚286690010767906 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚283107400097760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37831229 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3582270746712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19805821 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚230950642723328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚230950485135872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚227250571761664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚300 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66950028 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3589074310656 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37841386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚289690009967104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66939869 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚289690368279040 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚286101293968384 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 trainer/global_step‚3749 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚286101276449792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚289690368279040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3699937798656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29684546 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19805823 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚298 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29108640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3588733517312 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37831229 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 _runtime‚1894.791574385 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 $ _timestamp‚1.784488418216243e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29108642 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚286689653805112®X®§2È - _wandbruntime‚1894  _runtime‚1894z—á*É € 1loss/FBt1/assignment_loss‚0.06724810068547746 epoch‚4 % _timestamp‚1.7844884182183788e+09 _runtime‚1894.793216388 &loss/total_loss‚10.540385296361762 0loss/FRt1/assignment_loss‚1.1591667147643665 2loss/SRqqt1/assignment_loss‚0.5834195778304786 2loss/SRqqt2/assignment_loss‚0.6608136527800179 /loss/FRt1/detection_loss‚1.4362273664203093 1loss/SRqqt1/detection_loss‚0.6846476907382476 1loss/SRqqt2/detection_loss‚1.3232307211895074 /loss/FBt1/detection_loss‚0.7449263498634878 0loss/FRt2/assignment_loss‚2.2686701647739493 1loss/FBt2/assignment_loss‚0.00480575435172061 _step‚303 /loss/FBt2/detection_loss‚1.0592159955547407 -loss/FRt2/detection_loss‚0.54801320740946 trainer/global_step‚3749”ãf2 Ê … &loss/total_loss‚10.540385296361762 0loss/FRt1/assignment_loss‚1.1591667147643665 2loss/SRqqt1/assignment_loss‚0.5834195778304786 2loss/SRqqt2/assignment_loss‚0.6608136527800179 /loss/FRt1/detection_loss‚1.4362273664203093 1loss/SRqqt1/detection_loss‚0.6846476907382476 1loss/SRqqt2/detection_loss‚1.3232307211895074 /loss/FBt1/detection_loss‚0.7449263498634878 0loss/FRt2/assignment_loss‚2.2686701647739493 1loss/FBt2/assignment_loss‚0.00480575435172061 _step‚303 /loss/FBt2/detection_loss‚1.0592159955547407 -loss/FRt2/detection_loss‚0.54801320740946 trainer/global_step‚3749 1loss/FBt1/assignment_loss‚0.06724810068547746 epoch‚4 % _timestamp‚1.7844884182183788e+09 _runtime‚1894.793216388¯voHøUË ‚(:Ê ìÃôÒä“íË& disk.nvme1c1n1.out‚5751.1044921875$ disk.nvme1n1.out‚5751.1044921875" disk.dm-1.out‚3065.78271484375' disk.nvme0c0n1.out‚18950.3818359375 disk.nvme0n1.in‚ 4.3359375$ memory_percent‚96.97821140289307( proc.memory.availableMB‚ 495.08984375* proc.memory.percent‚17.363786697387695 cpu‚26.037211657506894 proc.cpu.threads‚17 network.sent‚ 4.933379e+06) disk./.usagePercent‚76.65103004802188# disk.dm-0.out‚21520.87255859375 disk.dm-1.in‚0 disk.nvme1n1.in‚0 network.recv‚540187 disk.dm-0.in‚ 4.3359375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375" proc.memory.rssMB‚ 2844.8828125% disk./.usageGB‚1176.7849388122559 disk.md0.in‚0 disk.md0.out‚53.6875 disk.rbd0.out‚0% disk.nvme0n1.out‚18950.3818359375 disk.nvme1c1n1.in‚0@ÛDÚŸÌ ‚(:” ìÃôÒ™†ŸÌ gpu.0.gpu‚68.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚120.251' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.083666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]ÀK2Í - _wandbruntime‚1919  _runtime‚1919;"AÉÎ Ã ' lr-AdamW/pg1‚0.00020979972888803125 ' lr-AdamW/pg2‚0.00020979972888803125 trainer/global_step‚3799 % _timestamp‚1.7844884431374743e+09 _runtime‚1919.712426785 _step‚304ÛøfØÎÏ È ' lr-AdamW/pg1‚0.00020979972888803125 ' lr-AdamW/pg2‚0.00020979972888803125 trainer/global_step‚3799 % _timestamp‚1.7844884431374743e+09 _runtime‚1919.712426785 _step‚304°¢èn­2Ð - _wandbruntime‚1919  _runtime‚1919 5W±Ñ ßO WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3634331427840 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38316956 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3627785588484 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3747095727104 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 _runtime‚1919.714923509 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚286705261099920 E=DeviceStatsMonitor.üzKìÆ"on_train_batch_start/segment.all.allocated‚934 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20054792 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚289736690653696 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚289736673135104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38327099 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3634672214016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3634672214016 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚293371362867712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚286705278139280 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3628125511806 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38316956 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚293371004562944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚230135512056320 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38327099 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚289736690653696 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30057494 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3634331427840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29476240 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20054790 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29476240 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 _step‚305 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67793194 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3746966751232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 trainer/global_step‚3799 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30057785 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚233882607783424 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚230135487631872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67803339 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚290333046688404 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚293371004562944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚290333403651086 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368¦ž?‹ê'Ò äO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29476238 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚289736690653696 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚289736673135104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38327099 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3634672214016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3634672214016 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚293371362867712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚286705278139280 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3628125511806 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38316956 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚293371004562944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚230135512056320 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38327099 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚289736690653696 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30057494 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3634331427840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29476240 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20054790 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29476240 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 _step‚305 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67793194 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3746966751232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 trainer/global_step‚3799 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30057785 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚233882607783424 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚230135487631872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67803339 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚290333046688404 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚293371004562944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚290333403651086 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3634331427840 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38316956 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3627785588484 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3747095727104 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 _runtime‚1919.714923509 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚286705261099920 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20054792 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38326831 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10005414 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚230194363964416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚290407757956914 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29483742 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚293446126243328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚295 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚233942287135232 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚230194388388864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7844884435160818e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3635261339136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67820730 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3748052139520 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3629053735154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3635602132480 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29483742 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29483740 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20059873 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚289810864904192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚293446484555264 _step‚306 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67810571 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30065287 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10005121 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290407400994120 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67820730 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚289810864904192 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚289810882422784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3635261339136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29483740 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30064992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚286778687182400 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67810571 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20059871 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3635602132480 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚06þ × îM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29483742 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚293446126243328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚295 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚233942287135232 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚230194388388864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7844884435160818e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3635261339136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67820730 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3748052139520 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3629053735154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3635602132480 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29483742 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29483740 E=Devic¥êeStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20059873 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚289810864904192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚293446484555264 _step‚306 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67810571 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30065287 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10005121 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290407400994120 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67820730 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚289810864904192 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚289810882422784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3635261339136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29483740 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30064992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚286778687182400 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67810571 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20059871 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3635602132480 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38326831 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10005414 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚230194363964416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚290407757956914²ÌÚÝ2Ø - _wandbruntime‚1920  _runtime‚1920J^¢Ù  2loss/FBt1/assignment_loss‚0.026470818794697273 /loss/FBt1/detection_loss‚0.4591149540404634 % _timestamp‚1.7844884435171626e+09 _runtime‚1920.091997664 2loss/SRqqt1/assignment_loss‚0.5733575050422295 1loss/SRqqt1/detection_loss‚0.6916734545403965 2loss/FBt2/assignment_loss‚0.045376613055073754 /loss/FRt1/detection_loss‚1.4198334215154105 /loss/FRt2/detection_loss‚0.5476842927479638 /loss/FBt2/detection_loss‚1.0435382484768827 _step‚307 0loss/SRqqt2/detection_loss‚1.040228184723381 %loss/total_loss‚9.616926898011753 epoch‚4 trainer/global_step‚3799 /loss/FRt1/assignment_loss‚0.975494873623888 0loss/FRt2/assignment_loss‚2.2389163490551622 2loss/SRqqt2/assignment_loss‚0.5552381823962051šGŠ Ú † 2loss/FBt2/assignment_loss‚0.045376613055073754 /loss/FRt1/detection_loss‚1.4198334215154105 /loss/FRt2/detection_loss‚0.5476842927479638 /loss/FBt2/detection_loss‚1.0435382484768827 _step‚307 0loss/SRqqt2/detection_loss‚1.040228184723381 %loss/total_loss‚9.616926898011753 epoch‚4 trainer/global_step‚3799 /loss/FRt1/assignment_loss‚0.975494873623888 0loss/FRt2/assignment_loss‚2.2389163490551622 2loss/SRqqt2/assignment_loss‚0.5552381823962051 2loss/FBt1/assignment_loss‚0.026470818794697273 /loss/FBt1/detection_loss‚0.4591149540404634 % _timestamp‚1.7844884435171626e+09 _runtime‚1920.091997664 2loss/SRqqt1/assignment_loss‚0.5733575050422295 1loss/SRqqt1/detection_loss‚0.6916734545403965³j†ËãZÛ ‚(:Ï ŠÄôÒ Ý»Ë# disk.dm-0.out‚21545.37451171875" disk.dm-1.out‚3066.11279296875 disk.nvme0c0n1.in‚ 4.3359375& disk.nvme0n1.out‚18975.22119140625" proc.memory.rssMB‚ 2844.8828125* proc.memory.percent‚17.363786697387695 disk.rbd0.out‚0$ memory_percent‚96.97713851928711 proc.cpu.threads‚17 network.sent‚ 5.048347e+06 network.recv‚548236 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0 disk.rbd0.in‚0 cpu‚26.033186133616056( disk./.usagePercent‚76.5892775472999' disk.nvme1c1n1.out‚5752.35009765625 disk.nvme1n1.in‚0 disk.md0.out‚ 54.25390625( disk.nvme0c0n1.out‚18975.22119140625 disk.nvme0n1.in‚ 4.3359375& proc.memory.availableMB‚ 495.265625% disk./.usageGB‚1175.8368835449219% disk.nvme1n1.out‚5752.350097656258héžÜ ‚(:“ ŠÄôÒ‚ؘÌ gpu.0.gpu‚89.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚54.0 gpu.0.powerWatts‚245.611' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚81.87033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€öù2Ý - _wandbruntime‚1945  _runtime‚1945mM:ÉÞ Ã ' lr-AdamW/pg1‚0.00020969767105377964 ' lr-AdamW/pg2‚0.00020969767105377964 trainer/global_step‚3849 % _timestamp‚1.7844884687168202e+09 _runtime‚1945.291804932 _step‚308NO Îß È _step‚308 ' lr-AdamW/pg1‚0.00020969767105377964 ' lr-AdamW/pg2‚0.00020969767105377964 trainer/global_step‚3849 % _timestamp‚1.7844884687168202e+09 _runtime‚1945.291804932´Ĉ õ2à - _wandbruntime‚1945  _runtime‚1945ieÀ|ã'á ÝO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚293446279108096 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚236874185343488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20308842 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3794881084416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29851340 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30438963 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3681200035840 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚294 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 _runtime‚1945.292950504 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚233079304259072 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚293446279108096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 trainer/global_step‚3849 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3795010060288 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68674041 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29851338 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚236874338743808 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚233079328683520 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10130123 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3674228653492 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29851340 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚297127479143936 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38812558 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚296 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚290376582263280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚290376565223920 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚309 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30439259 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38822701 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚297127120839168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚297127120839168 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 trainer/global_step‚3849 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3795010060288 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68674041 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29851338 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚236874338743808 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚233079328683520 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10130123 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3674228653492 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29851340 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚297127479143936 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38812558 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚296 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚290376582263280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚290376565223920 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚309 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30439259 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38822701 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚297127120839168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚297127120839168 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚294 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 _runtime‚1945.292950504 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚233079304259072 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚293446279108096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0µ°ÿ’¿2ã - _wandbruntime‚1945  _runtime‚1945¯Ñ‘ä èM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38832590 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚293520453358592 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38832590 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3795971129856 $ _timestamp‚1.784488469093275e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3681789160960 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374Ó‘so" SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29858840 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚297202600831488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚293520453358592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚1945.668475881 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3682129954304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30446587 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68681273 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚297202242519552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68691432 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚236934176145920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3682129954304 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38822433 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚293520470877184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚293520470877184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚290449991306400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3675496800162 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚290450008345760 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 trainer/global_step‚3849 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3681789160960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10132666 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚294125148183128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29858842 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29858842 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68691432 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20313923 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚233138205016064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3795837966848 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29858840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10132956 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38822433 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088›¯Ò„ó&å íM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚293520470877184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚290449991306400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3675496800162 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚290450008345760 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 trainer/global_step‚3849 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3681789160960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10132666 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚294125148183128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29858842 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29858842 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68691432 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20313923 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚233138205016064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3795837966848 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29858840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10132956 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38822433 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38832590 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚293520453358592 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38832590 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3795971129856 $ _timestamp‚1.784488469093275e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3681789160960 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29858840 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚297202600831488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚293520453358592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚1945.668475881 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3682129954304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30446587 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68681273 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚297202242519552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68691432 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚236934176145920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3682129954304 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38822433 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚293520470877184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259¶,f9`2æ - _wandbruntime‚1945  _runtime‚1945¶Lò¾ç € 1loss/SRqqt2/detection_loss‚1.0065582432394284 0loss/FBt1/detection_loss‚0.48228797973473203 epoch‚4 /loss/FBt2/detection_loss‚1.2172075255463521 /loss/FRt1/assignment_loss‚0.994844686225116 2loss/SRqqt1/assignment_loss‚0.6164586319294906 3loss/FBt2/assignment_loss‚0.0004679553718081782 /loss/FRt1/detection_loss‚1.5011654410834063 1loss/SRqqt1/detection_loss‚0.6347452390989992 % _timestamp‚1.7844884690944366e+09 _step‚311 .loss/FRt2/assignment_loss‚2.11112567849442 1loss/FBt1/assignment_loss‚0.07612786567399965 trainer/global_step‚3849 _runtime‚1945.669151042 %loss/total_loss‚9.911194969643327 2loss/SRqqt2/assignment_loss‚0.6522324921029923 .loss/FRt2/detection_loss‚0.617973231142582¹¿Ô è … /loss/FBt2/detection_loss‚1.2172075255463521 /loss/FRt1/assignment_loss‚0.994844686225116 2loss/SRqqt1/assignment_loss‚0.6164586319294906 3loss/FBt2/assignment_loss‚0.0004679553718081782 /loss/FRt1/detection_loss‚1.5011654410834063 1loss/SRqqt1/detection_loss‚0.6347452390989992 % _timestamp‚1.7844884690944366e+09 _step‚311 .loss/FRt2/assignment_loss‚2.11112567849442 1loss/FBt1/assignment_loss‚0.07612786567399965 trainer/global_step‚3849 _runtime‚1945.669151042 %loss/total_loss‚9.911194969643327 2loss/SRqqt2/assignment_loss‚0.6522324921029923 .loss/FRt2/detection_loss‚0.617973231142582 1loss/SRqqt2/detection_loss‚1.0065582432394284 0loss/FBt1/detection_loss‚0.48228797973473203 epoch‚4·ó³Yé ‚(:Î ™ÄôÒ†ðÈÌ disk.nvme0c0n1.in‚ 4.3359375& disk.nvme0n1.out‚18976.04541015625% disk.nvme1n1.out‚5753.02783203125" disk.dm-1.out‚3066.42724609375# memory_percent‚96.9773530960083" proc.memory.rssMB‚ 2844.8828125 cpu‚26.03073793868221( disk.nvme0c0n1.out‚18976.04541015625 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚5753.02783203125 disk.nvme1n1.in‚0* proc.memory.percent‚17.363786697387695 proc.cpu.threads‚17 network.recv‚552313) disk./.usagePercent‚76.58928599541632# disk.dm-0.out‚21545.97607421875 disk.dm-1.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 495.23046875 network.sent‚ 5.11206e+06 disk.nvme0n1.in‚ 4.3359375 disk.md0.in‚0 disk.md0.out‚ 54.51953125 disk.dm-0.in‚ 4.3359375 disk.rbd0.out‚0$ disk./.usageGB‚1175.837013244629+¦Vמê ‚(:“ ™ÄôÒφØË gpu.0.gpu‚59.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚232.583' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚77.52766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=kQ¡Që ‚(:Æ ¨ÄôÒìÀÔË% disk.nvme1n1.out‚5753.66259765625 disk.md0.out‚54.875 disk.dm-1.in‚0% disk.nvme0n1.out‚18978.5322265625 cpu‚26.02838241829879 network.sent‚ 5.161444e+06 disk.nvme1n1.in‚0 disk.dm-0.in‚ 4.3359375 disk.rbd0.out‚0( proc.memory.availableMB‚ 495.22265625" proc.memory.rssMB‚ 2844.8828125* proc.memory.percent‚17.363786697387695 proc.cpu.threads‚17 network.recv‚556555( disk./.usagePercent‚76.5892753110338 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚5753.66259765625 disk.md0.in‚0 disk.dm-0.out‚21548.18359375" disk.dm-1.out‚3066.57177734375' disk.nvme0c0n1.out‚18978.5322265625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375$ memory_percent‚96.97740077972412% disk./.usageGB‚1175.8368492126465¨!Kžì ‚(:“ ¨ÄôÒÀ³šÌ gpu.0.gpu‚57.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚228.554' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚76.18466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö‰”€2í - _wandbruntime‚1970  _runtime‚1970§þÄÈî  _runtime‚ 1970.85841156 _step‚312 ' lr-AdamW/pg1‚0.00020959395909214825 ' lr-AdamW/pg2‚0.00020959395909214825 trainer/global_step‚3899 % _timestamp‚1.7844884942834187e+09³÷’©Íï Ç ' lr-AdamW/pg1‚0.00020959395909214825 ' lr-AdamW/pg2‚0.00020959395909214825 trainer/global_step‚3899 % _timestamp‚1.7844884942834187e+09 _runtime‚ 1970.85841156 _step‚312¸ ¡gf2ð - _wandbruntime‚1970  _runtime‚1970Ê苲$ñ ßO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30226440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 trainer/global_step‚3899 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39318303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30820783 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚236023145310720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚300883237115392 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10257891 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124781568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39308160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚236023120886272 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30820497 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚297155867562496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20562890 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3721011641822 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3727727857664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3727387071488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3842992561664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300883595420160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10257607 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3842867780096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 _runtime‚1970.859514168 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69534598 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39318303 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3727387071488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚239865988666368 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚294047869347920 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 VHDeviceStatsMonitor.on_train_batch_start/inactive_Õd]êËsplit_bytes.all.current‚ 149206016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300883237115392 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30226440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7844884942844515e+09 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚239866137872384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20562892É­Üýê'ò äO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30820497 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚297155867562496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20562890 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3721011641822 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3727727857664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3727387071488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3842992561664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300883595420160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10257607 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3842867780096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 _runtime‚1970.859514168 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69534598 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39318303 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3727387071488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚239865988666368 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚294047869347920 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300883237115392 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30226440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7844884942844515e+09 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚239866137872384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20562892 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30226440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 trainer/global_step‚3899 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39318303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30820783 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚236023145310720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚300883237115392 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10257891 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124781568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39308160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚236023120886272¹­Ýž>2ó - _wandbruntime‚1971  _runtime‚1971L°ðåï&ô éM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30828423 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3728657776128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚300958717107712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 % _timestamp‚1.7844884946588633e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚300958717107712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39318035 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3721939865170 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30828133 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30233942 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 145004544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚236082021643264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 _runtime‚1971.234028646 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚239925828832256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39318035 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚297230059331584 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39328192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3843952193536 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10260162 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39328192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30233942 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69562134 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69551975 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10260450 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚314 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3728316982784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚297230041812992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69551975 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚236081997218816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 120580096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3721599941736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3728316982784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30233940 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20567971 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚294121312469760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚294121295430400 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3728657776128 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30233940 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20567973 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Nq2fô&õ îM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30828423 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3728657776128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚300958717107712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 % _timestamp‚1.7844884946588633e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚300958717107712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39318035 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3721939865170 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30828133 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30233942 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 145004544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚236082021643264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 _runtime‚1971.234028646 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚239925828832256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39318035 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚297230059331584 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39328192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3843952193536 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10260162 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39328192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30233942 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69562134 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69551975 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10260450 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚314 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3728316982784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚297230041812992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69551975 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚236081997218816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 120580096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3721599941736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3728316982784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30233940 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20567971 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚294121312469760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚294121295430400 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3728657776128 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30233940 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20567973 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0º1D5á2ö - _wandbruntime‚1971  _runtime‚1971fW±÷ ‚ 1loss/SRqqt2/detection_loss‚0.8555252654477954 &loss/total_loss‚10.203669668588274 _step‚315 0loss/FRt2/assignment_loss‚2.0720355335122838 0loss/FBt1/assignment_loss‚0.1087860699761726 0loss/SRqqt1/detection_loss‚0.651684645980575 % _timestamp‚1.7844884946604588e+09 _runtime‚1971.235252518 0loss/FRt1/assignment_loss‚1.0081692879376973 2loss/SRqqt1/assignment_loss‚0.6381907810977219 2loss/SRqqt2/assignment_loss‚0.7811662411983233 2loss/FBt2/assignment_loss‚0.008482307060863345 0loss/FBt1/detection_loss‚0.37453005082098506 /loss/FBt2/detection_loss‚1.4405814063336169 trainer/global_step‚3899 /loss/FRt1/detection_loss‚1.6479211709742723 /loss/FRt2/detection_loss‚0.6165969082479666 epoch‚4V sÄø ‡ 0loss/FRt1/assignment_loss‚1.0081692879376973 2loss/SRqqt1/assignment_loss‚0.6381907810977219 2loss/SRqqt2/assignment_loss‚0.7811662411983233 2loss/FBt2/assignment_loss‚0.008482307060863345 0loss/FBt1/detection_loss‚0.37453005082098506 /loss/FBt2/detection_loss‚1.4405814063336169 trainer/global_step‚3899 /loss/FRt1/detection_loss‚1.6479211709742723 /loss/FRt2/detection_loss‚0.6165969082479666 epoch‚4 1loss/SRqqt2/detection_loss‚0.8555252654477954 &loss/total_loss‚10.203669668588274 _step‚315 0loss/FRt2/assignment_loss‚2.0720355335122838 0loss/FBt1/assignment_loss‚0.1087860699761726 0loss/SRqqt1/detection_loss‚0.651684645980575 †…I % _timestamp‚1.7844884946604588e+09 _runtime‚1971.235252518»>ær,Yù ‚(:Î ·ÄôÒßÎõË! disk.dm-1.out‚3067.1396484375 disk.rbd0.out‚0& disk.nvme0n1.out‚19439.22802734375( proc.memory.availableMB‚ 495.16015625 cpu‚26.026281044774972 network.sent‚ 5.180197e+06" proc.memory.rssMB‚ 2844.8828125* proc.memory.percent‚17.363786697387695 proc.cpu.threads‚17) disk./.usagePercent‚76.64310298162454% disk./.usageGB‚1176.6632385253906% disk.nvme1n1.out‚5786.09033203125 disk.dm-0.in‚ 4.3359375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚19439.22802734375 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚5786.09033203125 network.recv‚559915 disk.nvme1n1.in‚0 disk.md0.out‚ 55.21484375 disk.dm-0.out‚22040.01171875 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375$ memory_percent‚96.97778224945068 disk.md0.in‚0&6|žú ‚(:“ ·ÄôÒÜðªÌ gpu.0.gpu‚90.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚120.832' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.27733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 :Sû ‚(:È ÆÄôÒ©Ó¸Ë proc.cpu.threads‚17 disk.md0.out‚ 55.7421875 network.sent‚ 5.228878e+06" disk.nvme0n1.out‚ 20049.7734375' disk.nvme1c1n1.out‚5787.27392578125# disk.dm-0.out‚22650.39697265625) disk./.usagePercent‚76.64333232313739 disk.nvme0n1.in‚ 4.3359375% disk.nvme1n1.out‚5787.27392578125 disk.dm-1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚17.363786697387695 network.recv‚564062% disk./.usageGB‚1176.6667594909668$ disk.nvme0c0n1.out‚ 20049.7734375 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375! disk.dm-1.out‚3067.3583984375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0$ memory_percent‚96.97778224945068( proc.memory.availableMB‚ 495.16015625" proc.memory.rssMB‚ 2844.8828125 cpu‚26.023982206875115DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3774255679488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10384849 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3773914893312 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚300865456016896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚300865456016896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 _runtime‚1996.420552159 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚304639353391616 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304639353391616 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚304639711696384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 % _timestamp‚1.7844885198455112e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39803762 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3773914893312 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30601538 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 _step‚317 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚238966937513472 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31201791 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3891025268224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3890896292352 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚238966961937920 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚301486645218110 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3774255679488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚297719190511280 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚297719173471920 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368;¡¸1ê'‚äO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304639353391616 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚304639711696384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 % _timestamp‚1.7844885198455112e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39803762 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3773914893312 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30601538 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 _step‚317 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚238966937513472 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31201791 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3891025268224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3890896292352 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚238966961937920 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚301486645218110 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3774255679488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚297719190511280 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚297719173471920 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30601540 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3767114783508 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3774255679488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10384849 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3773914893312 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚300865456016896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚300865456016896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 _runtime‚1996.420552159 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚304639353391616 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 trainer/global_step‚3949 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚304714475072000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3774844804608 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39813637 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31209388 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39823794 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚300939647785984 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10387365 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚301560642561144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3774844804608 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70422677 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20822021 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30609042 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3775185597952 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70432836 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39813637 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10387080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚242917674127360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 _step‚318 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3768043006744 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30609042 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30609040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30609040 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚300939630267392 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31209101 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122677248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3891860281344 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30609040 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚300939630267392 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31209101 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122677248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3891860281344 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 trainer/global_step‚3949 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚304714475072000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3774844804608 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39813637 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31209388 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39823794 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚300939647785984 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10387365 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚301560642561144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3774844804608 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70422677 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20822021 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30609042 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3775185597952 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70432836 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39813637 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10387080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚242917674127360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368¾ªzVè2†- _wandbruntime‚1996  _runtime‚1996Ù·e‡ %loss/total_loss‚9.959302086254954 epoch‚4 0loss/FRt2/assignment_loss‚2.2290656082408944 4loss/FBt2/assignment_loss‚0.00030659117931594105 /loss/FBt2/detection_loss‚0.7665354532792288 _step‚319 1loss/SRqqt1/assignment_loss‚0.662244142056871 1loss/FBt1/assignment_loss‚0.12982071124694056 /loss/FRt2/detection_loss‚0.5976074746241625 trainer/global_step‚3949 $ _timestamp‚1.784488520221932e+09 /loss/FRt1/assignment_loss‚1.095280752517283 /loss/FRt1/detection_loss‚1.4600537837650336 1loss/SRqqt1/detection_loss‚0.6456597415496892 1loss/SRqqt2/detection_loss‚0.9835898097689179 /loss/FBt1/detection_loss‚0.7487748869487246 _runtime‚1996.797485634 2loss/SRqqt2/assignment_loss‚0.6403631310778918¢òM² ˆ† /loss/FRt1/assignment_loss‚1.095280752517283 /loss/FRt1/detection_loss‚1.4600537837650336 1loss/SRqqt1/detection_loss‚0.6456597415496892 1loss/SRqqt2/detection_loss‚0.9835898097689179 /loss/FBt1/detection_loss‚0.7487748869487246 _runtime‚1996.797485634 2loss/SRqqt2/assignment_loss‚0.6403631310778918 %loss/total_loss‚9.959302086254954 epoch‚4 0loss/FRt2/assignment_loss‚2.2290656082408944 4loss/FBt2/assignment_loss‚0.00030659117931594105 /loss/FBt2/detection_loss‚0.7665354532792288 _step‚319 1loss/SRqqt1/assignment_loss‚0.662244142056871 1loss/FBt1/assignment_loss‚0.12982071124694056 /loss/FRt2/detection_loss‚0.5976074746241625 trainer/global_step‚3949 $ _timestamp‚1.784488520221932e+09¿õ…þY‰‚(:Î ÕÄôÒàÉ’Ì" proc.memory.rssMB‚ 2844.8828125 disk.md0.out‚ 56.06640625! disk.dm-1.out‚3067.5693359375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1n1.in‚0 proc.cpu.threads‚17 network.sent‚ 5.247628e+06 network.recv‚567286 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚22650.96337890625 disk.dm-1.in‚0' disk.nvme0c0n1.out‚20050.6220703125( proc.memory.availableMB‚ 495.12109375* proc.memory.percent‚17.363786697387695% disk./.usageGB‚1176.6667594909668% disk.nvme0n1.out‚20050.6220703125' disk.nvme1c1n1.out‚5787.91552734375% disk.nvme1n1.out‚5787.91552734375 cpu‚26.02168562585501) disk./.usagePercent‚76.64333232313739 disk.md0.in‚0 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0$ memory_percent‚96.97802066802979fÀ‚¤žŠ‚(:“ ÕÄôÒùÁÄÌ gpu.0.gpu‚58.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚55.0 gpu.0.powerWatts‚242.659' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚80.88633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚÈ‘2‹- _wandbruntime‚2021  _runtime‚2021*ë2 Ɍà ' lr-AdamW/pg1‚0.00020938157945784425 ' lr-AdamW/pg2‚0.00020938157945784425 trainer/global_step‚3999 % _timestamp‚1.7844885450733771e+09 _runtime‚2021.648353994 _step‚320ÁA.¾ÎÈ trainer/global_step‚3999 % _timestamp‚1.7844885450733771e+09 _runtime‚2021.648353994 _step‚320 ' lr-AdamW/pg1‚0.00020938157945784425 ' lr-AdamW/pg2‚0.00020938157945784425ÀF™cN2Ž- _wandbruntime‚2021  _runtime‚2021g’øå'ßO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71276002 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30976640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3813897771838 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31582417 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚304575044471296 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40309507 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21070990 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚241910754140672 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚308395469667840 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚321 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7844885450743706e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚305204035444436 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚304575026952704 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40299364 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3938889120768 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚304575044471296 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71286147 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚305204392407118 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚308395827972608 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚308395469667840 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚304575026952704 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚241910778565120 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3820442715136 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚308395827972608 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3820783501312 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3820442715136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40309507 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚301390494635280 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚298 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚245849643261440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30976638 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21070992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10511725 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3820442715136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40309507 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚301390494635280 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚298 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚245849643261440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30976638 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21070992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10511725 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚241910754140672 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚308395469667840 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚321 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7844885450743706e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚305204035444436 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚304575026952704 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40299364 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3938889120768 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚304575044471296 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71286147 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 N@DeviceStatsMonitor.on_train_batch_start/a‰ÂÏÌctive_bytes.all.current‚ 358304768 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚305204392407118 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚308395827972608 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚308395469667840 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚304575026952704 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚241910778565120 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3820442715136 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚308395827972608 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675Á®õñ[2‘- _wandbruntime‚2022  _runtime‚2022ê›àÏï&’éM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71293379 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 trainer/global_step‚3999 _step‚322 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30984140 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚308470949660160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚245909636646400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71293379 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚304649218721792 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71303538 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31590308 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 _runtime‚2022.024467085 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21076073 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10514235 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10513945 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31590016 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3939850682880 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3821372626432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3814486071752 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3821713419776©É6ô&“îM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 _runtime‚2022.024467085 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21076073 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10514235 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10513945 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31590016 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3939850682880 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3821372626432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3814486071752 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3821713419776 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71293379 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 trainer/global_step‚3999 _step‚322 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30984140 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚308470949660160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚245909636646400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71293379 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚304649218721792 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71303538 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31590308Â2lZ„2”- _wandbruntime‚2022  _runtime‚2022ÝJR•ÿ 0loss/FRt2/assignment_loss‚2.1992612557601086 .loss/FRt1/detection_loss‚1.399844118488545 /loss/FBt2/detection_loss‚1.1083648037165403 %loss/total_loss‚9.999589627950673 epoch‚4 trainer/global_step‚3999 _runtime‚2022.025173706 _step‚323 1loss/SRqqt1/assignment_loss‚0.527444133691054 2loss/FBt1/assignment_loss‚0.025229654662834243 0loss/SRqqt2/detection_loss‚1.148328086404179 0loss/FRt1/assignment_loss‚0.8931822560261935 /loss/FRt2/detection_loss‚0.6428450054047335 1loss/SRqqt1/detection_loss‚0.7224562399684422 % _timestamp‚1.7844885454503994e+09 2loss/SRqqt2/assignment_loss‚0.6112395342091699 1loss/FBt2/assignment_loss‚0.10400308113289247 /loss/FBt1/detection_loss‚0.6173914584859803ºü0; –„ 2loss/SRqqt2/assignment_loss‚0.6112395342091699 1loss/FBt2/assignment_loss‚0.10400308113289247 /loss/FBt1/detection_loss‚0.6173914584859803 0loss/FRt2/assignment_loss‚2.1992612557601086 .loss/FRt1/detection_loss‚1.399844118488545 /loss/FBt2/detection_loss‚1.1083648037165403 %loss/total_loss‚9.999589627950673 epoch‚4 trainer/global_step‚3999 _runtime‚2022.025173706 _step‚323 1loss/SRqqt1/assignment_loss‚0.527444133691054 2loss/FBt1/assignment_loss‚0.025229654662834243 0loss/SRqqt2/detection_loss‚1.148328086404179 0loss/FRt1/assignment_loss‚0.8931822560261935 /loss/FRt2/detection_loss‚0.6428450054047335 1loss/SRqqt1/detection_loss‚0.7224562399684422 % _timestamp‚1.7844885454503994e+09ÃÚWJbU—‚(:Ê äÄôÒ–ÜË$ proc.memory.availableMB‚494.8125 proc.cpu.threads‚17 network.recv‚571802% disk./.usageGB‚1176.6667289733887 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 disk.nvme1n1.in‚0% disk.nvme1n1.out‚5790.58544921875 disk.md0.in‚0" proc.memory.rssMB‚ 2844.8828125 network.sent‚ 5.296051e+06" disk.dm-0.out‚22666.4541015625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.md0.out‚ 56.43359375* proc.memory.percent‚17.363786697387695 cpu‚26.01969825652812 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0( disk.nvme0c0n1.out‚20064.35693359375& disk.nvme0n1.out‚20064.35693359375 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚5790.58544921875$ memory_percent‚96.97990417480469( disk./.usagePercent‚76.6433303353453! disk.dm-1.out‚3067.6826171875¢=>ž˜‚(:“ äÄôÒôÞõË gpu.0.gpu‚91.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚130.739' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚43.57966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´ [™‚(:Ð óÄôÒ’’…Ì disk.rbd0.out‚0' disk.nvme0c0n1.out‚20065.5068359375 disk.nvme0n1.in‚ 4.3359375 disk.nvme1n1.in‚0 disk.md0.out‚ 56.78515625$ memory_percent‚97.01626300811768 disk.dm-0.in‚ 4.3359375 disk.md0.in‚0( proc.memory.availableMB‚ 488.85546875 proc.cpu.threads‚17! disk.dm-1.out‚3067.8369140625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0* proc.memory.percent‚17.400050163269043 cpu‚26.037195459284256 network.sent‚ 5.360148e+06 network.recv‚576052# disk.dm-0.out‚22667.27587890625 disk.dm-1.in‚0% disk.nvme0n1.out‚20065.5068359375' disk.nvme1c1n1.out‚5791.18505859375% disk.nvme1n1.out‚5791.18505859375# proc.memory.rssMB‚ 2850.82421875) disk./.usagePercent‚76.64333356550745% disk./.usageGB‚1176.66677856445317µ¯žš‚(:“ óÄôÒïîºÌ gpu.0.gpu‚34.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚117.928' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.309333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GiÐ2›- _wandbruntime‚2043  _runtime‚2043/*'eœýz /particle/accuracy_4_of_4‚0.9521350860595703 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 jet/accuracy_2_of_5‚1 jet/accuracy_3_of_5‚1 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99846351146698 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5074974298477173 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5520833730697632 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9936695098876953 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 epoch‚4 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14430734515190125 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5606434941291809 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6961846351623535 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.012743906117975712 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7589706182479858 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16053621470928192 _runtime‚2043.841132659 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03606702759861946 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.020114246755838394 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14279866218566895 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7361258864402771 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.09408508986234665 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.660023033618927 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14024792611598969 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9821428656578064 (loss/val_total_loss‚9.17270389800357 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9982993602752686 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12378957122564316 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7346282005310059 +particle/sensitivity‚0.6196476221084595 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8633986115455627 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9196552038192749 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16229908168315887 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17530585825443268 &particle/f_score‚0.638864278793335 7validation_average_jet_accuracy‚0.34158194065093994 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1863037794828415 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6944189667701721 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.05547479912638664 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3635064959526062 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12429961562156677 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8008770942687988 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1432700902223587 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6158019304275513 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 jet/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23684470355510712 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6636572480201721 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.16240182518959045 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.18287087976932526 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8058896660804749 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12083333730697632 particle/accuracy_1_of_3‚1 jet/accuracy_5_of_5‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13996535539627075 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2777777910232544 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.16373278200626373 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5992363691329956 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6246756315231323 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_1_of_1‚0.2936863601207733 particle/accuracy_1_of_2‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6524372100830078 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.43490010499954224 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7012053728103638 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7925890684127808 +jet/accuracy_3_of_3‚0.06718941777944565 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18053802847862244 particle/accuracy_3_of_5‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6169511675834656 particle/accuracy_3_of_3‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02079940214753151 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6984923481941223 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4856770634651184 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.723931610584259 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.863645076751709 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6884Ö{ª¢$766221046448 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4380028247833252 particle/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6202013492584229 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6202013492584229 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14967533946037292 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.40181317925453186 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4980708658695221 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787886738777161 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.05240795016288757 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7337350249290466 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7593508958816528 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0357142873108387 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5556315183639526 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9908856153488159 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9958078265190125 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.27633970975875854 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.49685773253440857 % _timestamp‚1.7844885672656589e+09 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9978183507919312 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11305470764636993 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7042677998542786 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_3‚0.9267746806144714 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.949999988079071 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9966425895690918 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6791493892669678 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5590842366218567 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.800000011920929 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9970434904098511 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7217166423797607 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05366162583231926 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.38271602988243103 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9978060126304626 *jet/accuracy_2_of_4‚0.7451327443122864 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7816239595413208 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.40500327944755554 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15032479166984558 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5946765542030334 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15224617719650269 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 _step‚324 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1292448788881302 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5786837339401245 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4166663587093353 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6447857022285461 jet/accuracy_4_of_5‚0 5 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.99006587266922 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14967533946037292 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_4‚1 (particle/accuracy‚0.8495898246765137 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10000000149011612 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9957410097122192 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.031003886833786964 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26130521297454834 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_1‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 *jet/accuracy_1_of_2‚0.7682587504386902 +particle/specificity‚0.9124638438224792 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0357142873108387 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15429939329624176 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.22057966887950897 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.055883295834064484 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45500001311302185 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.615227222442627 particle/accuracy_2_of_4‚1 /particle/accuracy_3_of_4‚0.9965986609458923 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16027477383613586 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7811660766601562 particle/accuracy_5_of_5‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4015151858329773 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1368032544851303 ,jet/accuracy_4_of_4‚0.014682539738714695 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15312205255031586 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.620673656463623 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6634043455123901 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_2_of_2‚1 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3605237603187561 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6552186012268066 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4287939965724945 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9970434904098511 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.05366162583231926 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5603934526443481 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7272883653640747 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893944263458252 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7921051979064941 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8850367665290833 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9948211908340454 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7925890684127808 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9241360425949097 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6493266224861145 particle/accuracy_2_of_3‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9871794581413269 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5600712895393372 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9969671368598938 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.800000011920929 +jet/accuracy_2_of_2‚0.13933059573173523 *jet/accuracy_1_of_4‚0.9578978419303894 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6377086043357849 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8"Purity/2FRt0SRqqt0FBt/event_purity‚0.022731963545084 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10000000149011612 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888193011283875 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9958513975143433 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.05296393111348152 trainer/global_step‚4004 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9961152076721191 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0™oˆ=‚{ :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02079940214753151 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6984923481941223 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4856770634651184 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.723931610584259 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.863645076751709 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6884766221046448 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4380028247833252 particle/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6202013492584229 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6202013492584229 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14967533946037292 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.40181317925453186 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4980708658695221 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787886738777161 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.05240795016288757 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7337350249290466 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7593508958816528 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0357142873108387 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5556315183639526 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9908856153488159 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9958078265190125 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.27633970975875854 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.49685773253440857 % _timestamp‚1.7844885672656589e+09 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9978183507919312 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11305470764636993 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7042677998542786 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_3‚0.9267746806144714 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.949999988079071 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9966425895690918 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6791493892669678 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5590842366218567 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.800000011920929 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9970434904098511 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7217166423797607 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05366162583231926 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.38271602988243103 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9978060126304626 *jet/accuracy_2_of_4‚0.7451327443122864 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7816239595413208 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.40500327944755554 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15032479166984558 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5946765542030334 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15224617719650269 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 _step‚324 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1292448788881302 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5786837339401245 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4166663587093353 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6447857022285461 jet/accuracy_4_of_5‚0 5 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.99006587266922 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14967533946037292 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_4‚1 (particle/accuracy‚0.8495898246765137 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10000000149011612 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9957410097122192 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.031003886833786964 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26130521297454834 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_1‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 *jet/accuracy_1_of_2‚0.7682587504386902 +particle/specificity‚0.9124638438224792 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0357142873108387 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15429939329624176 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.22057966887950897 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.055883295834064484 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45500001311302185 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.615227222442627 particle/accuracy_2_of_4‚1 /particle/accuracy_3_of_4‚0.9965986609458923 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16027477383613586 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7811660766601562 particle/accuracy_5_of_5‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4015151858329773 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1368032544851303 ,jet/accuracy_4_of_4‚0.014682539738714695 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15312205255031586 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.620673656463623 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6634043455123901 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_2_of_2‚1 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3605237603187561 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6552186012268066 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4287939965724945 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9970434904098511 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.05366162583231926 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5603934526443481 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7272883653640747 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893944263458252 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7921051979064941 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8850367665290833 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9948211908340454 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7925890684127808 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9241360425949097 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6493266224861145 particle/accuracy_2_of_3‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9871794581413269 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5600712895393372 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9969671368598938 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.800000011920929 +jet/accuracy_2_of_2‚0.13933059573173523 *jet/accuracy_1_of_4‚0.9578978419303894 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6377086043357849 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8"Purity/2FRt0SRqqt0FBt/event_purity‚0.022731963545084 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10000000149011612 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9888193011283875 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9958513975143433 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.05296393111348152 trainer/global_step‚4004 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9961152076721191 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 /particle/accuracy_4_of_4‚0.9521350860595703 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 jet/accuracy_2_of_5‚1 jet/accuracy_3_of_5‚1 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99846351146698 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5074974298477173 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5520833730697632 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9936695098876953 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 epoch‚4 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14430734515190125 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5606434941291809 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6961846351623535 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.012743906117975712 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7589706182479858 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16053621470928192 _runtime‚2043.841132659 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03606702759861946 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.020114246755838394 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14279866218566895 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7361258864402771 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.09408508986234665 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.660023033618927 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14024792611598969 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9821428656578064 (loss/val_total_loss‚9.17270389800357 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9982993602752686 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12378957122564316 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7346282005310059 +particle/sensitivity‚0.6196476221084595 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8633986115455627 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9196552038192749 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16229908168315887 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17530585825443268 &particle/f_score‚0.638864278793335 7validation_average_jet_accuracy‚0.34158194065093994 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1863037794828415 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6944189667701721 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.05547479912638664 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3635064959526062 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12429961562156677 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8008770942687988 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1432700902223587 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6158019304275513 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 jet/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23684470355510712 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6636572480201721 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.16240182518959045 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.18287087976932526 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8058896660804749 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12083333730697632 particle/accuracy_1_of_3‚1 jet/accuracy_5_of_5‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13996535539627075 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2777777910232544 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.16373278200626373 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5992363691329956 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6246756315231323 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_1_of_1‚0.2936863601207733 particle/accuracy_1_of_2‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6524372100830078 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.43490010499954224 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7012053728103638 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7925890684127808 +jet/accuracy_3_of_3‚0.06718941777944565 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18053802847862244 particle/accuracy_3_of_5‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6169511675834656 particle/accuracy_3_of_3‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1ÄÊ´Q›Xž‚(:Í ‚ÅôÒ䇖Ì( disk./.usagePercent‚76.6433641278109% disk./.usageGB‚1176.6672477722168 disk.nvme0n1.in‚ 4.3359375% disk.nvme0n1.out‚20068.3466796875$ memory_percent‚95.47412395477295% disk.nvme1n1.out‚6037.39794921875 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 network.sent‚ 5.374996e+06 disk.nvme1n1.in‚0 disk.md0.out‚ 57.1640625 disk.dm-0.in‚ 4.3359375! disk.dm-1.out‚3313.2939453125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0 network.recv‚578657# disk.dm-0.out‚22670.04736328125' disk.nvme0c0n1.out‚20068.3466796875' disk.nvme1c1n1.out‚6037.39794921875( proc.memory.availableMB‚ 741.51953125" proc.memory.rssMB‚ 2852.1171875* proc.memory.percent‚17.407941818237305 cpu‚26.038855343163462/M´­ŸŸ‚(:” ‚ÅôÒƒ ÈÌ gpu.0.gpu‚90.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚52.0 gpu.0.powerWatts‚120.463' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.154333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T]m‹2 - _wandbruntime‚2067  _runtime‚2067 &YÆ¡À $ _timestamp‚1.784488591162965e+09 _runtime‚2067.737959323 _step‚325 & lr-AdamW/pg1‚0.0002092729151868625 & lr-AdamW/pg2‚0.0002092729151868625 trainer/global_step‚4049¡Ç1»Ë¢Å & lr-AdamW/pg1‚0.0002092729151868625 & lr-AdamW/pg2‚0.0002092729151868625 trainer/global_step‚4049 $ _timestamp‚1.784488591162965e+09 _runtime‚2067.737959323 _step‚325Å *ôÁ2£- _wandbruntime‚2067  _runtime‚2067M¢†/¤ÞO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21511520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚326 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72561583 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚296 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚314123092652544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72551438 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4011871279616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31592902 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚294 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚314123450957312 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚314123450957312 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.cZÚ¹µurrent‚ 340786176 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3891486592000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚310863700763436 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚310863343800754 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40958536 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32246128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚250459194969088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40968679 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784488591164019e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21511522 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40958536 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31592904 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10734606 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚246447456859648 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚306978852883760 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3884490916994 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 trainer/global_step‚4049 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚314123092652544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚246447481284096í1?é'¥ãO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31592902 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚310231606060544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32245832 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚310231623579136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚310231606060544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4011738109440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3891486592000 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40968679 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10734312 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚250459352563712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3884830840316 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚296 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚314123092652544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72551438 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4011871279616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31592902 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚294 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚314123450957312 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚314123450957312 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3891486592000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚310863700763436 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚310863343800754 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40958536 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32246128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚250459194969088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40968679 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784488591164019e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21511522 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40958536 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31592904 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10734606 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚246447456859648 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚306978852883760 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3884490916994 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 trainer/global_step‚4049 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚314123092652544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚246447481284096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21511520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚326 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72561583 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3885419140230 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚246506357616640 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31600406 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40978568 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚250519026549760 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3892416503296 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72578974 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚310937698106470 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10737042 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4012824423424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40978568 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚314198572644864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚314198214332928 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32253351 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚250519182040064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40968411 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚307052296005600 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 _step‚327 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3892757296640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72568815 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21516603 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32253645 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚310305815348224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 trainer/global_step‚4049 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚307052278966240 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3885759063664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3892416503296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10736750 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31600404 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 _runtime‚2068.116299787¾ÉÛ ¨îM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚314198572644864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚314198214332928 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32253351 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚250519182040064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40968411 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚307052296005600 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 _step‚327 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3892757296640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72568815 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21516603 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32253645 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚310305815348224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 trainer/global_step‚4049 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚307052278966240 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3885759063664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3892416503296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10736750 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31600404 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 _runtime‚2068.116299787 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3892757296640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚310305797829632 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3885419140230 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚246506357616640 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31600406 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40978568 XDDevì»ÚÓÖiceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚250519026549760 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3892416503296 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72578974 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚310937698106470 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10737042 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4012824423424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40978568ÇéròI2©- _wandbruntime‚2068  _runtime‚2068nLZÁª 2loss/SRqqt1/assignment_loss‚0.5708093003057475 3loss/FBt2/assignment_loss‚0.0005957547464774393 1loss/SRqqt1/detection_loss‚0.7942513884363775 1loss/SRqqt2/detection_loss‚1.0474739007231502 /loss/FBt2/detection_loss‚0.7884755879640579 %loss/total_loss‚9.656089441408202 0loss/FRt1/assignment_loss‚0.9524499688128876 .loss/FRt1/detection_loss‚1.586677006846332 epoch‚5 _runtime‚2068.120422625 /loss/FRt2/assignment_loss‚2.272353748971052 2loss/SRqqt2/assignment_loss‚0.5159282171847405 /loss/FRt2/detection_loss‚0.6090496828880603 .loss/FBt1/detection_loss‚0.499377108731174 trainer/global_step‚4049 % _timestamp‚1.7844885915422347e+09 2loss/FBt1/assignment_loss‚0.018647775798144903 _step‚328ÄЋ «† 1loss/SRqqt1/detection_loss‚0.7942513884363775 1loss/SRqqt2/detection_loss‚1.0474739007231502 /loss/FBt2/detection_loss‚0.7884755879640579 %loss/total_loss‚9.656089441408202 0loss/FRt1/assignment_loss‚0.9524499688128876 .loss/FRt1/detection_loss‚1.586677006846332 epoch‚5 _runtime‚2068.120422625 /loss/FRt2/assignment_loss‚2.272353748971052 2loss/SRqqt2/assignment_loss‚0.5159282171847405 /loss/FRt2/detection_loss‚0.6090496828880603 .loss/FBt1/detection_loss‚0.499377108731174 trainer/global_step‚4049 % _timestamp‚1.7844885915422347e+09 2loss/FBt1/assignment_loss‚0.018647775798144903 _step‚328 2loss/SRqqt1/assignment_loss‚0.5708093003057475 3loss/FBt2/assignment_loss‚0.0005957547464774393ÈR؇T¬‚(:É ‘ÅôÒ„÷ÒË* proc.memory.percent‚17.407941818237305 cpu‚26.03643002087267& disk.nvme0c0n1.out‚20281.919921875 disk.nvme0n1.in‚ 4.3359375% disk.nvme1n1.out‚6313.86669921875 disk.md0.in‚0& proc.memory.availableMB‚ 741.453125" proc.memory.rssMB‚ 2852.1171875 proc.cpu.threads‚17% disk./.usageGB‚1176.9049987792969 disk.rbd0.out‚0$ memory_percent‚95.47452926635742 network.recv‚582697) disk./.usagePercent‚76.65885027059416 disk.dm-0.in‚ 4.3359375 disk.dm-1.out‚3558.873046875 disk.nvme0c0n1.in‚ 4.3359375$ disk.nvme0n1.out‚20281.919921875' disk.nvme1c1n1.out‚6313.86669921875 disk.nvme1n1.in‚0 network.sent‚ 5.419113e+06# disk.dm-0.out‚22913.90283203125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0 disk.md0.out‚ 57.44140625ƒò€­‚(:’ ‘ÅôÒ­¥®Ì gpu.0.gpu‚31.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚261.947' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.31566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡‚ÿU®‚(:Ê  ÅôÒªæÿË& disk.nvme0c0n1.out‚20286.185546875 disk.nvme0n1.in‚ 4.3359375( proc.memory.availableMB‚ 741.44921875 network.sent‚ 5.483301e+06 network.recv‚586932' disk.nvme1c1n1.out‚6320.94384765625% disk.nvme1n1.out‚6320.94384765625 disk.md0.out‚ 58.0390625$ memory_percent‚95.47455310821533* proc.memory.percent‚17.407941818237305) disk./.usagePercent‚76.65882318692688 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.rbd0.in‚0" proc.memory.rssMB‚ 2852.1171875 cpu‚26.034135674352132 proc.cpu.threads‚17$ disk./.usageGB‚1176.904582977295# disk.dm-0.out‚22923.86474609375 disk.dm-1.out‚3558.982421875 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.3359375$ disk.nvme0n1.out‚20286.185546875 disk.nvme1n1.in‚0 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0ÛašêŸ¯‚(:”  ÅôÒœö³Ì gpu.0.gpu‚64.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚52.0 gpu.0.powerWatts‚117.993' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.330999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X6¿ƒ2°- _wandbruntime‚2092  _runtime‚2092ƒamɱà ' lr-AdamW/pg1‚0.00020916260359182833 ' lr-AdamW/pg2‚0.00020916260359182833 trainer/global_step‚4099 % _timestamp‚1.7844886163753076e+09 _runtime‚2092.951430845 _step‚329å¼½Î²È ' lr-AdamW/pg1‚0.00020916260359182833 ' lr-AdamW/pg2‚0.00020916260359182833 trainer/global_step‚4099 % _timestamp‚1.7844886163753076e+09 _runtime‚2092.951430845 _step‚329É,A&É2³- _wandbruntime‚2092  _runtime‚2092o­ý†ä'´ÞO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31968002 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32624989 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4059874574848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚330 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚73422140 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚313941212033536 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚253451172486144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚317879208928768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3938355200000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3938014413824 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41454138 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚317879208928768 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32625282 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21765570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41454138 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚313941212033536 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31968004 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚317879567233536 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31968004 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3931273905324 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚317879567233536 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3938014413824 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73432285 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31968002 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313941194514944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚249391297911296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10859710 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41464281 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176èNA3é'µãO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚313941194514944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚313941212033536 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚253451172486144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚317879208928768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3938355200000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3938014413824 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41454138 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚317879208928768 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32625282 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21765570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41454138 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚313941212033536 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31968004 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚317879567233536 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31968004 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3931273905324 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚317879567233536 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3938014413824 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73432285 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31968002 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313941194514944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚249391297911296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10859710 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41464281 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31968002 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32624989 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4059874574848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚330 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚73422140 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3932202128672 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚314655802258272 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚310723583090240 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3931862205238 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73439517 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10861962 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚249450149819392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3939285118464 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 _runtime‚2093.325924106 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32632613 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41474170 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4060699662336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/numi§ CY_oom_rejections‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚310723600129600 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31975506 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31975504 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 trainer/global_step‚4099 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73449676 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7844886167507467e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31975504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10862245 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21770653 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚317954688921088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚317954330609152 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚314655445295478 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21770651 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚314015386284032 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚314015386284032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3938944325120 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚249450174243840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73449676 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2596‘)ô&¸îM TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41474170 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41464013 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3932202128672 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚314655802258272 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚310723583090240 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3931862205238 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73439517 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10861962 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚249450149819392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3939285118464 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 _runtime‚2093.325924106 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32632613 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41474170 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4060699662336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚310723600129600 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31975506 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31975504 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 trainer/global_step‚4099 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73449676 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7844886167507467e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31975504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10862245 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21770653 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚317954688921088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚317954330609152 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚314655445295478 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21770651 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚314015386284032 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚314015386284032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3938944325120 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚249450174243840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73449676 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4060826533888 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚314015403802624 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚253511000777728 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0Ë…`¿2¹- _wandbruntime‚2093  _runtime‚20930B‰‡ºü _runtime‚ 2093.32763339 _step‚332 0loss/FBt2/assignment_loss‚0.1517087019510548 /loss/FRt1/detection_loss‚1.5860398789077261 /loss/FRt2/detection_loss‚0.5405657372235739 #loss/total_loss‚9.8044927195495 % _timestamp‚1.7844886167526386e+09 0loss/FRt1/assignment_loss‚1.1415964023720833 2loss/SRqqt2/assignment_loss‚0.5904917885568524 1loss/SRqqt1/detection_loss‚0.6272643665617256 /loss/FBt1/detection_loss‚0.5804795742421447 trainer/global_step‚4099 /loss/FRt2/assignment_loss‚2.331555167957369 2loss/SRqqt1/assignment_loss‚0.6315385609418664 2loss/FBt1/assignment_loss‚0.026305409865155126 0loss/SRqqt2/detection_loss‚0.846974939703234 /loss/FBt2/detection_loss‚0.7499721912667155 epoch‚5&Cé0» #loss/total_loss‚9.8044927195495 % _timestamp‚1.7844886167526386e+09 0loss/FRt1/assignment_loss‚1.1415964023720833 2loss/SRqqt2/assignment_loss‚0.5904917885568524 1loss/SRqqt1/detection_loss‚0.6272643665617256 /loss/FBt1/detection_loss‚0.5804795742421447 trainer/global_step‚4099 /loss/FRt2/assignment_loss‚2.331555167957369 2loss/SRqqt1/assignment_loss‚0.6315385609418664 2loss/FBt1/assignment_loss‚0.026305409865155126 0loss/SRqqt2/detection_loss‚0.846974939703234 /loss/FBt2/detection_loss‚0.7499721912667155 epoch‚5 _runtime‚ 2093.32763339 _step‚332 0loss/FBt2/assignment_loss‚0.1517087019510548 /loss/FRt1/detection_loss‚1.5860398789077261 /loss/FRt2/detection_loss‚0.5405657372235739Ì$ô‘àž¼‚(:“ ¯ÅôÒÉý²Ì gpu.0.gpu‚91.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚262.327' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.44233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Mšý´W½‚(:Ì ¯ÅôÒß…øÌ' disk.nvme1c1n1.out‚6321.74169921875 disk.nvme1n1.in‚0$ memory_percent‚95.47626972198486( proc.memory.availableMB‚ 741.16796875* proc.memory.percent‚17.407941818237305 network.recv‚590475 disk.rbd0.in‚0% disk.nvme0n1.out‚20287.1025390625 cpu‚26.03209525211118 network.sent‚ 5.486324e+06# disk.dm-0.out‚22924.42724609375 disk.nvme0c0n1.in‚ 4.3359375 disk.md0.out‚ 58.45703125" proc.memory.rssMB‚ 2852.1171875 proc.cpu.threads‚17) disk./.usagePercent‚76.64334251057187 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚6321.74169921875 disk.md0.in‚0% disk./.usageGB‚1176.6669158935547 disk.dm-1.out‚ 3559.25390625' disk.nvme0c0n1.out‚20287.1025390625ĨìwT¾‚(:É ¾ÅôÒ’¾ÕË disk.rbd0.out‚0 disk.nvme0n1.in‚ 4.3359375 disk.md0.out‚ 58.98046875" disk.dm-0.out‚22937.4033203125$ memory_percent‚95.47486305236816" proc.memory.rssMB‚ 2852.1171875' disk.nvme1c1n1.out‚6324.47119140625 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375' proc.memory.availableMB‚ 741.3984375 proc.cpu.threads‚17) disk./.usagePercent‚76.62784841662024$ disk./.usageGB‚1176.429042816162 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.nvme0c0n1.in‚ 4.3359375( disk.nvme0c0n1.out‚20298.54638671875* proc.memory.percent‚17.407941818237305 cpu‚26.02990530458906 network.sent‚ 5.550401e+06 network.recv‚594465& disk.nvme0n1.out‚20298.54638671875% disk.nvme1n1.out‚6324.47119140625 disk.dm-1.in‚0 disk.dm-1.out‚ 3559.328125 disk.rbd0.in‚0¼w–kŸ¿‚(:” ¾ÅôÒ¸ŒÌ gpu.0.gpu‚57.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚52.0 gpu.0.powerWatts‚118.459' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.486333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëc²2À- _wandbruntime‚2118  _runtime‚2118sÙÞuÉÁà % _timestamp‚1.7844886414555845e+09 _runtime‚2118.031418997 _step‚333 ' lr-AdamW/pg1‚0.00020905064643960593 ' lr-AdamW/pg2‚0.00020905064643960593 trainer/global_step‚4149€¢“ÎÂÈ trainer/global_step‚4149 % _timestamp‚1.7844886414555845e+09 _runtime‚2118.031418997 _step‚333 ' lr-AdamW/pg1‚0.00020905064643960593 ' lr-AdamW/pg2‚0.00020905064643960593Ítœúø2Ã- _wandbruntime‚2118  _runtime‚2118 n¿å'ÄßO P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚314321478171120 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32343102 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41949740 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚321635325204992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4107838636544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚314321461131760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41949740 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124781568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33004925 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32343102 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3977716970332 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10985024 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚256442803969024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10985303 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚252335114538496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚321635325204992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚317650800487936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚317650782969344 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74302987 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚318299195141452 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74292842 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41959883 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚317650800487936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚321635683509760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚321635683509760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3984883021824 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33004925 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32343102 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3977716970332 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10985024 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚256442803969024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10985303 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚252335114538496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚321635325204992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚317650800487936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚317650782969344 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74302987 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚318299195141452 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74292842 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41959883 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚317650800487936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚321635683509760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚321635683509760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3984883021824 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74310219 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚318373192484486 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 % _timestamp‚1.7844886418251574e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚252393990871040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚335 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚256502643253248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚317724974738432 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33012208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32350606 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74310219 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4108803678208 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚314394887214240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22024703 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3978645193680 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚317724992257024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3985812940288 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚317724992257024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3985812940288 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22024701 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41959615 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚256502794549248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3985472146944 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41969772 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚321710805197312 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚321710805197312 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74320378 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3978305270246 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3985472146944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41969772¶>e1ô&ÈîM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚318373192484486 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 % _timestamp‚1.7844886418251574e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚252393990871040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚335 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚256502643253248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚317724974738432 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33012208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32350606 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74310219 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4108803678208 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚314394887214240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22024703 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3978645193680 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚317724992257024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3985812940288 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚317724992257024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3985812940288 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22024701 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41959615 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚256502794549248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3985472146944 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41969772 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚321710805197312 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚321710805197312 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74320378 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3978305270246 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3985472146944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41969772 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 trainer/global_step‚4149 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74310219Ï ©Üœ2É- _wandbruntime‚2118  _runtime‚2118+‡' ʃ .loss/FRt1/detection_loss‚1.401068900424857 /loss/FRt2/detection_loss‚0.5796712169454378 1loss/SRqqt1/detection_loss‚0.7030578143908092 0loss/FRt1/assignment_loss‚1.0585278755472611 2loss/SRqqt1/assignment_loss‚0.5446830718719028 2loss/SRqqt2/assignment_loss‚0.8219531430576227 1loss/SRqqt2/detection_loss‚0.9012126848691061 0loss/FBt1/detection_loss‚0.42853882027786133 %loss/total_loss‚9.216619965833397 /loss/FBt2/detection_loss‚0.6263292991245787 epoch‚5 trainer/global_step‚4149 _runtime‚2118.400813187 _step‚336 2loss/FBt1/assignment_loss‚0.014881867471091647 % _timestamp‚1.7844886418261418e+09 .loss/FRt2/assignment_loss‚2.13653840211207 4loss/FBt2/assignment_loss‚0.00015686974079850795l Íˈ 0loss/FRt1/assignment_loss‚1.0585278755472611 2loss/SRqqt1/assignment_loss‚0.5446830718719028 2loss/SRqqt2/assignment_loss‚0.8219531430576227 1loss/SRqqt2/detection_loss‚0.9012126848691061 0loss/FBt1/detection_loss‚0.42853882027786133 %loss/total_loss‚9.216619965833397 /loss/FBt2/detection_loss‚0.6263292991245787 epoch‚5 trainer/global_step‚4149 _runtime‚2118.400813187 _step‚336 2loss/FBt1/assignment_loss‚0.014881867471091647 % _timestamp‚1.7844886418261418e+09 .loss/FRt2/assignment_loss‚2.13653840211207 4loss/FBt2/assignment_loss‚0.00015686974079850795 .loss/FRt1/detection_loss‚1.401068900424857 /loss/FRt2/detection_loss‚0.5796712169454378 1loss/SRqqt1/detection_loss‚0.7030578143908092ÐÕðkòSÌ‚(:È ÍÅôÒ»±ÈË disk.nvme0n1.in‚ 4.3359375 disk.nvme1n1.in‚0% disk./.usageGB‚1176.4295082092285 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚6325.494140625$ memory_percent‚95.47510147094727& proc.memory.availableMB‚ 741.359375 network.recv‚597883 disk.rbd0.out‚0( disk.nvme0c0n1.out‚20299.61962890625& disk.nvme0n1.out‚20299.61962890625# disk.nvme1n1.out‚6325.494140625* proc.memory.percent‚17.407941818237305 cpu‚26.028416017323053 proc.cpu.threads‚17 network.sent‚ 5.553365e+06 disk.md0.out‚59.40625" disk.dm-1.out‚3559.55126953125 disk.rbd0.in‚0" proc.memory.rssMB‚ 2852.1171875) disk./.usagePercent‚76.62787873044967 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375" disk.dm-0.out‚22938.3564453125 disk.nvme0c0n1.in‚ 4.3359375áoœ“Í‚(:ˆ ÍÅôÒú‚¨Ì gpu.0.gpu‚70.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚52.0 gpu.0.powerWatts‚117.981' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.327 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fYBÓ2Î- _wandbruntime‚2143  _runtime‚214358=¯ÈÏ ' lr-AdamW/pg1‚0.00020893704552341625 ' lr-AdamW/pg2‚0.00020893704552341625 trainer/global_step‚4199 % _timestamp‚1.7844886666300194e+09 _runtime‚ 2143.20500093 _step‚337©K9ÍÐÇ ' lr-AdamW/pg1‚0.00020893704552341625 ' lr-AdamW/pg2‚0.00020893704552341625 trainer/global_step‚4199 % _timestamp‚1.7844886666300194e+09 _runtime‚ 2143.20500093 _step‚337Ñþ)pð2Ñ- _wandbruntime‚2143  _runtime‚2143~É‘/¾ ÒßO IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚325391799785984 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33383905 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4155657127936 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4023820112018 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32718202 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚322016585367778 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4155792395264 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 eQDeviceStì]ý'atsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚255278931165696 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4024160035340 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚325391799785984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4031410843648 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75173689 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11110528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚322016942330460 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22273670 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32718204 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4031070057472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11110235 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32718202 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚255278906741248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚317992765255760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32718204 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33384200 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚338 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚325391441481216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 135267328 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 trainer/global_step‚4199 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75163544 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4031070057472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚325391441481216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22273672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4031410843648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 _runtime‚2143.205782691 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42455485 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚259434723560960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159691776 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42455485 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42445342 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚321360371423744 % _timestamp‚1.7844886666309428e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚321360371423744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934Ž)Ïúê'ÓäO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4031410843648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 _runtime‚2143.205782691 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42455485 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚259434723560960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159691776 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42455485 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42445342 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚321360371423744 % _timestamp‚1.7844886666309428e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚321360371423744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚325391799785984 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33383905 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4155657127936 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4023820112018 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32718202 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚322016585367778 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4155792395264 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚255278931165696 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4024160035340 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚325391799785984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4031410843648 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75173689 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11110528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚322016942330460 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22273670 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32718204 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4031070057472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11110235 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32718202 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚255278906741248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚317992765255760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32718204 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33384200 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚338 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚325391441481216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 135267328 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 trainer/global_step‚4199 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75163544 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4031070057472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚325391441481216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22273672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664Òb°Û/2Ô- _wandbruntime‚2143  _runtime‚2143üÄ(õï&ÕéM _step‚339 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚321434563192832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11113031 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚318066191338240 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22278753 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42465374 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33391487 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚259494400692736 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75191080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚325466921473536 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚325466563161600 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4031999968768 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4156617618944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4032340762112 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 _runtime‚2143.575028256 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42455217 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4024748335254 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42465374 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11112736 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚322090939673494 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75191080 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32725704 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32725704 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32725706 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚322091296636288 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚325466921473536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚321434563192832 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4032340762112 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 _runtime‚2143.575028256 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42455217 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4024748335254 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42465374 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11112736 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚322090939673494 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75191080 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32725704 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32725704 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 ”ðêZIC=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32725706 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚322091296636288 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚325466921473536 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚321434563192832 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚318066191338240 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22278753 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42465374 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33391487 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚259494400692736 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75191080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚325466921473536 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚325466563161600 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4031999968768 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4156617618944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F‰Ý‚(:’ ëÅôÒš¥£Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚52.0 gpu.0.powerWatts‚117.754' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.251333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤¯82Þ- _wandbruntime‚2168  _runtime‚2168¢QÜuÉßà trainer/global_step‚4249 % _timestamp‚1.7844886922209234e+09 _runtime‚2168.795994638 _step‚341 ' lr-AdamW/pg1‚0.00020882180266280863 ' lr-AdamW/pg2‚0.00020882180266280863:hÏ&ÎàÈ trainer/global_step‚4249 % _timestamp‚1.7844886922209234e+09 _runtime‚2168.795994638 _step‚341 ' lr-AdamW/pg1‚0.00020882180266280863 ' lr-AdamW/pg2‚0.00020882180266280863Õ7j“ñ2á- _wandbruntime‚2168  _runtime‚2168ßdàä'âÞO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚329147916062208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚262426491364352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚258222747792896 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚325069977396736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚325069959878144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33764969 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚329147557757440 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚321664069379760 trainer/global_step‚4249 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚329147916062208 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42951087 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚262426340061184 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚2168.800369145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42940944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76044391 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚321664086419120 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4070263177026 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42951087 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4077597879296 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚325734689519468 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4203743571456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4077938665472 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33093304 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33764686 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11236966 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42940944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 _step‚342 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚329147557757440 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76034246 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22527722 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚325069959878144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4203616692736 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4077938665472 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 $ _timestamp‚1.784488692223154e+09 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚325069977396736 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚325734332556786 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4070603100348 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33093304 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33764686 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11236966 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42940944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 _step‚342 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚329147557757440 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76034246 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22527722 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚325069959878144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4203616692736 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4077938665472 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 $ _timestamp‚1.784488692223154e+09 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚325069977396736 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚325734332556786 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4070603100348 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚329147557757440 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚321664069379760 trainer/global_step‚4249Ö{øX<2ä- _wandbruntime‚2169  _runtime‚2169õBäD* åèM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76051623 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4204575563776 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76061782 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚343 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11239407 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 E=DeviceStatsMonitor.on_train_batc¦2SsÄh_end/segment.small_pool.peak‚259 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33100806 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚329223037749760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚325144151647232 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4204706629632 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4078527790592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚262486175264768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42950819 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚329222679437824 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329223037749760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33100806 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 trainer/global_step‚4249 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4078868583936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22532801 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚325808686862502 $ _timestamp‚1.784488692596246e+09 _runtime‚2169.171381243 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76061782 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚258281624125440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚321737495462240 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33100804 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚321737512501600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚295 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4078527790592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33100804 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4071191400262 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42960976 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚325809043825296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33772208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76061782 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚343 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11239407 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33100806 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚329223037749760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚325144151647232 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4204706629632 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4078527790592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚262486175264768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42950819 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚329222679437824 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329223037749760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33100806 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 trainer/global_step‚4249 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4078868583936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22532801 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚325808686862502 $ _timestamp‚1.784488692596246e+09 _runtime‚2169.171381243 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76061782 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚258281624125440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚321737495462240 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33100804 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚321737512501600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚295 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4078527790592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33100804 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4071191400262 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42960976 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚325809043825296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33772208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76051623 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088×Áv2ç- _wandbruntime‚2169  _runtime‚2169ÎvÉ è -loss/FBt1/detection_loss‚5.04485335341737 /loss/FBt2/detection_loss‚0.9313532100059092 0loss/FRt2/assignment_loss‚2.0921120290116315 2loss/SRqqt2/assignment_loss‚0.7583520569725696 3loss/FBt2/assignment_loss‚0.0028401147174541563 trainer/global_step‚4249 1loss/FBt1/assignment_loss‚0.11622699754524529 /loss/FRt2/detection_loss‚0.5686345666214053 _runtime‚2169.174879129 2loss/SRqqt1/assignment_loss‚0.6479089484676452 1loss/SRqqt1/detection_loss‚0.6779042781869482 &loss/total_loss‚14.248903770782064 epoch‚5 % _timestamp‚1.7844886925972934e+09 _step‚344 0loss/FRt1/assignment_loss‚0.8865540213735679 .loss/FRt1/detection_loss‚1.435234720837104 1loss/SRqqt2/detection_loss‚1.0869294736252137ø!‡ é† epoch‚5 % _timestamp‚1.7844886925972934e+09 _step‚344 0loss/FRt1/assignment_loss‚0.8865540213735679 .loss/FRt1/detection_loss‚1.435234720837104 1loss/SRqqt2/detection_loss‚1.0869294736252137 -loss/FBt1/detection_loss‚5.04485335341737 /loss/FBt2/detection_loss‚0.9313532100059092 0loss/FRt2/assignment_loss‚2.0921120290116315 2loss/SRqqt2/assignment_loss‚0.7583520569725696 3loss/FBt2/assignment_loss‚0.0028401147174541563 trainer/global_step‚4249 1loss/FBt1/assignment_loss‚0.11622699754524529 /loss/FRt2/detection_loss‚0.5686345666214053 _runtime‚2169.174879129 2loss/SRqqt1/assignment_loss‚0.6479089484676452 1loss/SRqqt1/detection_loss‚0.6779042781869482 &loss/total_loss‚14.248903770782064جÓ+sRê‚(:Ç úÅôÒ¢ßÌÌ disk.dm-1.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚20477.8212890625% disk.nvme1n1.out‚6423.69482421875 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375# disk.dm-0.out‚23211.80517578125" proc.memory.availableMB‚741.25" proc.memory.rssMB‚ 2852.1171875* proc.memory.percent‚17.407941818237305 cpu‚26.042196742773807 disk.nvme1c1n1.in‚0$ disk./.usageGB‚1176.667568206787! disk.dm-1.out‚3559.9365234375 disk.nvme0c0n1.in‚ 4.3359375' disk.nvme0c0n1.out‚20477.8212890625$ memory_percent‚95.47576904296875 proc.cpu.threads‚17 network.recv‚609803) disk./.usagePercent‚76.64338499962788 disk.rbd0.in‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 60.61328125' disk.nvme1c1n1.out‚6423.69482421875 network.sent‚ 5.69566e+06Ûªúœë‚(:‘ úÅôÒ•·àË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚52.0 gpu.0.powerWatts‚117.547' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.18233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]¯ïŒTì‚(:É ‰ÆôÒ–ûÎË$ disk.nvme1c1n1.out‚ 6424.19140625* proc.memory.percent‚17.407941818237305 network.recv‚614021 disk.nvme0c0n1.in‚ 4.3359375' disk.nvme0c0n1.out‚20478.6025390625% disk.nvme0n1.out‚20478.6025390625" disk.nvme1n1.out‚ 6424.19140625 disk.md0.out‚ 60.88671875 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚23212.34423828125 network.sent‚ 5.747038e+06 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1176.6291275024414" disk.dm-1.out‚3560.05810546875 disk.md0.in‚0# memory_percent‚96.9799280166626( proc.memory.availableMB‚ 494.80859375" proc.memory.rssMB‚ 2852.1171875 cpu‚26.047605655072417) disk./.usagePercent‚76.64088112701133 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0ë8*5í‚(:’ ‰ÆôÒ˜þ‰Ì gpu.0.gpu‚94.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚133.42' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚44.47333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ã3$2î- _wandbruntime‚2193  _runtime‚2193ÞÓuzÆïÀ trainer/global_step‚4299 $ _timestamp‚1.784488716566692e+09 _runtime‚2193.141636306 _step‚345 & lr-AdamW/pg1‚0.0002087049197036313 & lr-AdamW/pg2‚0.0002087049197036313mÒËðÅ & lr-AdamW/pg1‚0.0002087049197036313 & lr-AdamW/pg2‚0.0002087049197036313 trainer/global_step‚4299 $ _timestamp‚1.784488716566692e+09 _runtime‚2193.141636306 _step‚345Ù[g2ñ- _wandbruntime‚2193  _runtime‚2193ÁVE¾å'òßO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43436546 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7844887165673473e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚332903674033664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22781772 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚265418125234688 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚325335390543120 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11363163 _runtime‚2193.145428506 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚332903674033664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4124125701120 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34145228 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4251585239040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4124466487296 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 trainer/global_step‚4299 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33468402 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚328779565851136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚261166539995648 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4251720506368 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚325335373503760 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33468404 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚329452436708476 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76904948 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚328779565851136 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159691776 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4124466487296 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚332904032338432 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚332904032338432 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33468404 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76915093 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚328779548332544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4124125701120 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚328779548332544 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43436546 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4116706242034 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚261166564420096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 _step‚346 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34144933 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22781770 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11363456 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚265418284926464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43446689 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4117046165356 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43446689 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 135267328 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 IÅÅ&itor.on_train_batch_start/segment.small_pool.current‚259 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33468404 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76915093 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚328779548332544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4124125701120 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚328779548332544 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43436546 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4116706242034 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚261166564420096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 _step‚346 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34144933 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22781770 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11363456 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚265418284926464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43446689 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4117046165356 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43446689 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 135267328 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚332903674033664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4124125701120 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34145228 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4251585239040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4124466487296 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 trainer/global_step‚4299 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33468402 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚328779565851136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚261166539995648 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4251720506368 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚325335373503760 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33468404 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚329452436708476 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76904948 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚328779565851136 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159691776 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4124466487296 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚332904032338432 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚332904032338432 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448ÚÇþªØ2ô- _wandbruntime‚2193  _runtime‚2193¼ì`Zï&õéM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2193.811347747 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4125396405760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11365971 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚328853757620224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22786851 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚329526434051510 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚265478116217344 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4117634465270 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4252675464704 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚329526791014304 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4125396405760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚332979154025984 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33475906 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43456578 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11365680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76922325 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4117974388704 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚332978795714048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34152531 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43446421 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43456578 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33475904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚265477962824192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚325408799586240 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 % _timestamp‚1.7844887172361887e+09 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚328853757620224 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22786853 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚347 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76932484 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33475904 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚332979154025984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76932484 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33475906 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43446421 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76922325 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 trainer/global_step‚4299 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚328853740101632 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚261225440752640 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2193.811347747 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4125396405760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11365971 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚328853757620224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22786851 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚329526434051510 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚265478116217344 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4117634465270 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4252675464704 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚329526791014304 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4125396405760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚332979154025984 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33475906 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43456578 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11365680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76922325 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4117974388704 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚332978795714048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34152531 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43446421 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43456578 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33475904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚265477962824192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚325408799586240 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 % _timestamp‚1.7844887172361887e+09 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚328853757620224 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22786853 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚347 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76932484 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259Û³‰3’2÷- _wandbruntime‚2193  _runtime‚2193‰ÃOtø 0loss/FRt2/assignment_loss‚2.1443727111865973 2loss/FBt1/assignment_loss‚0.011981298752510883 1loss/FBt2/assignment_loss‚0.06684014061795057 .loss/FRt1/detection_loss‚1.663394811891036 1loss/SRqqt2/detection_loss‚1.0669865248973172 /loss/FBt1/detection_loss‚0.7361458183029697 $loss/total_loss‚9.69680420840588 trainer/global_step‚4299 0loss/FRt1/assignment_loss‚1.0126058256145016 /loss/FRt2/detection_loss‚0.5413909206180961 % _timestamp‚1.7844887172372136e+09 _runtime‚2193.812082179 2loss/SRqqt1/assignment_loss‚0.6204882261444531 0loss/FBt2/detection_loss‚0.45368938471959985 epoch‚5 2loss/SRqqt2/assignment_loss‚0.7156153816365016 1loss/SRqqt1/detection_loss‚0.6632931640243455 _step‚348 Ú † ù† 2loss/SRqqt1/assignment_loss‚0.6204882261444531 0loss/FBt2/detection_loss‚0.45368938471959985 epoch‚5 2loss/SRqqt2/assignment_loss‚0.7156153816365016 1loss/SRqqt1/detection_loss‚0.6632931640243455 _step‚348 0loss/FRt2/assignment_loss‚2.1443727111865973 2loss/FBt1/assignment_loss‚0.011981298752510883 1loss/FBt2/assignment_loss‚0.06684014061795057 .loss/FRt1/detection_loss‚1.663394811891036 1loss/SRqqt2/detection_loss‚1.0669865248973172 /loss/FBt1/detection_loss‚0.7361458183029697 $loss/total_loss‚9.69680420840588 trainer/global_step‚4299 0loss/FRt1/assignment_loss‚1.0126058256145016 /loss/FRt2/detection_loss‚0.5413909206180961 % _timestamp‚1.7844887172372136e+09 _runtime‚2193.812082179Ü÷0IbSú‚(:È ˜ÆôÒý°›Ì* proc.memory.percent‚17.407941818237305# disk.nvme1n1.out‚6464.810546875" disk.dm-1.out‚3560.17138671875' disk.nvme0c0n1.out‚20576.1318359375 network.sent‚ 5.762804e+06% disk./.usageGB‚1176.8703002929688 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚76.65659015098315 disk.md0.out‚ 61.609375 disk.dm-0.in‚ 4.3359375 disk.nvme0c0n1.in‚ 4.3359375$ memory_percent‚96.98019027709961" proc.memory.rssMB‚ 2852.1171875 cpu‚26.053240547998957 proc.cpu.threads‚17 disk.dm-1.in‚0 network.recv‚617625% disk.nvme1c1n1.out‚6464.810546875# disk.dm-0.out‚23348.85986328125 disk.nvme0n1.in‚ 4.3359375% disk.nvme0n1.out‚20576.1318359375& proc.memory.availableMB‚ 494.765625ý㟓û‚(:ˆ ˜ÆôÒçðÕÌ gpu.0.gpu‚72.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚248.937' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚82.979 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ,Øæ2ü- _wandbruntime‚2217  _runtime‚2217 óg9ýà _runtime‚2217.589379404 _step‚349 ' lr-ФøAdamW/pg1‚0.00020858639851800215 ' lr-AdamW/pg2‚0.00020858639851800215 trainer/global_step‚4349 % _timestamp‚1.7844887410144234e+091:÷ÎþÈ ' lr-AdamW/pg1‚0.00020858639851800215 ' lr-AdamW/pg2‚0.00020858639851800215 trainer/global_step‚4349 % _timestamp‚1.7844887410144234e+09 _runtime‚2217.589379404 _step‚349ÝŠ[A¬2ÿ- _wandbruntime‚2217  _runtime‚2217“*çä'€ÞO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4163489230364 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 $ _timestamp‚1.784488741015307e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚332489154305536 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4299665031168 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚336659790309888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚333169826934802 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚336660148614656 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4170653522944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4170653522944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 _runtime‚2217.590165155 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚336659790309888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚336660148614656 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4163149307042 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33843504 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34525736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚264110381047296 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11490193 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚332489154305536 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚264110356622848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43932148 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23035822 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23035820 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33843504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11489916 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚329006677627760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34526015 trainer/global_step‚4349 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33843502 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚268410046078464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0FªÐ_é'ãO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23035820 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33843504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11489916 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚329006677627760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34526015 trainer/global_step‚4349 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33843502 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚268410046078464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4163489230364 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 $ _timestamp‚1.784488741015307e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚332489154305536 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4299665031168 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚336659790309888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚333169826934802 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚336660148614656 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4170653522944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4170653522944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 _runtime‚2217.590165155 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚336659790309888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚336660148614656 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4163149307042 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33843504 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34525736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚264110381047296 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11490193 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚332489154305536 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚264110356622848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43932148 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23035822 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768Þ%Za’2‚- _wandbruntime‚2217  _runtime‚2217Ó£k5î&ƒèM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚264169257379840 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚336735270302208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23040901 % _timestamp‚1.7844887413724506e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚336735270302208 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4171924227584 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4171583434240 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚268469880531456 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚329080120749600 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77803186 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11492387 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚268469731332608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4300623151616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23040903 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77793027 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚332563346074624 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 _runtime‚ 2217.94767016 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77793027 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43942023 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34533288 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43952180 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚336734911990272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚332563346074624 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚329080103710240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚332563328556032 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 _runtime‚ 2217.94767016 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77793027 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43942023 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34533288 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43952180 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚336734911990272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚332563346074624 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚329080103710240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚332563328556032 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚268469880531456 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚329080120749600 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77803186 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11492387 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚268469731332608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4300623151616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088ßÞ› ˆ2…- _wandbruntime‚2217  _runtime‚2217H–¬<†þ 1loss/FBt2/assignment_loss‚0.04212803899928872 /loss/FBt2/detection_loss‚1.1370179893821477 _runtime‚2217.948213171 /loss/FRt1/detection_loss‚1.4178957843589557 1loss/SRqqt1/detection_loss‚0.5801613981918986 1loss/SRqqt2/detection_loss‚1.1327444080301132 /loss/FBt1/detection_loss‚0.4584881374156694 %loss/total_loss‚9.529189305893453 % _timestamp‚1.7844887413733795e+09 /loss/FRt2/assignment_loss‚2.203846329567479 2loss/SRqqt2/assignment_loss‚0.6445513230900286 trainer/global_step‚4349 _step‚352 epoch‚5 /loss/FRt1/assignment_loss‚0.699463190902181 1loss/SRqqt1/assignment_loss‚0.626454828662503 /loss/FRt2/detection_loss‚0.5720635826845842 1loss/FBt1/assignment_loss‚0.01437429460860411ò3{* ‡ƒ 1loss/SRqqt2/detection_loss‚1.1327444080301132 /loss/FBt1/detection_loss‚0.4584881374156694 %loss/total_loss‚9.529189305893453 % _timestamp‚1.7844887413733795e+09 /loss/FRt2/assignment_loss‚2.203846329567479 2loss/SRqqt2/assignment_loss‚0.6445513230900286 trainer/global_step‚4349 _step‚352 epoch‚5 /loss/FRt1/assignment_loss‚0.699463190902181 1loss/SRqqt1/assignment_loss‚0.626454828662503 /loss/FRt2/detection_loss‚0.5720635826845842 1loss/FBt1/assignment_loss‚0.01437429460860411 1loss/FBt2/assignment_loss‚0.04212803899928872 /loss/FBt2/detection_loss‚1.1370179893821477 _runtime‚2217.948213171 /loss/FRt1/detection_loss‚1.4178957843589557 1loss/SRqqt1/detection_loss‚0.5801613981918986àÀ mWˆ‚(:Ì §ÆôÒ·¹÷Ë disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚6517.58056640625% disk.nvme1n1.out‚6517.58056640625 disk.md0.out‚ 61.8203125' disk.nvme0c0n1.out‚20593.9931640625$ memory_percent‚96.98264598846436 cpu‚26.058261712589886 network.recv‚621552 disk.dm-1.out‚3560.333984375% disk.nvme0n1.out‚20593.9931640625" proc.memory.rssMB‚ 2852.1171875 proc.cpu.threads‚17 network.sent‚ 5.825007e+06 disk.nvme1n1.in‚0 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚23418.86376953125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375( proc.memory.availableMB‚ 494.36328125) disk./.usagePercent‚76.65636577895052 disk.md0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 4.3359375* proc.memory.percent‚17.407941818237305$ disk./.usageGB‚1176.866855621338‰Ž´}ž‰‚(:“ §ÆôÒéµ´Ì gpu.0.gpu‚91.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚120.578' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.19266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VÜ¡TŠ‚(:É ¶ÆôÒË% disk./.usageGB‚1176.8669776916504 disk.md0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.3359375& disk.nvme0c0n1.out‚20657.701171875' disk.nvme1c1n1.out‚6520.59912109375" proc.memory.rssMB‚ 2852.1171875 network.recv‚626113( disk./.usagePercent‚76.6563737301189 disk.rbd0.in‚0 disk.nvme0n1.in‚ 4.3359375# memory_percent‚96.9825267791748* proc.memory.percent‚17.407941818237305 cpu‚26.060775234570315 disk.nvme1n1.in‚0 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚23484.53369140625 disk.dm-1.in‚0$ disk.nvme0n1.out‚20657.701171875' proc.memory.availableMB‚ 494.3828125 proc.cpu.threads‚17 network.sent‚ 5.874494e+06% disk.nvme1n1.out‚6520.59912109375 disk.md0.out‚ 62.22265625 disk.dm-1.out‚3560.509765625 disk.nvme1c1n1.in‚0Ãã ™ž‹‚(:“ ¶ÆôÒâ«ïË gpu.0.gpu‚52.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚54.0 gpu.0.powerWatts‚265.342' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.44733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó…`2Œ- _wandbruntime‚2243  _runtime‚2243 ø@È trainer/global_step‚4399 $ _timestamp‚1.784488766646878e+09 _runtime‚2243.222683928 _step‚353 ' lr-AdamW/pg1‚0.00020846624100427852 ' lr-AdamW/pg2‚0.00020846624100427852ß‘ˆ³ÍŽÇ _runtime‚2243.222683928 _step‚353 ' lr-AdamW/pg1‚0.00020846624100427852 ' lr-AdamW/pg2‚0.00020846624100427852 trainer/global_step‚4399 $ _timestamp‚1.784488766646878e+09á‡òùU2- _wandbruntime‚2243  _runtime‚2243^?™ºå'ßO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚340416264890880 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚336198742759936 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚271401861072896 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34218602 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚4399 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 % _timestamp‚1.7844887666477158e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4217522130944 _step‚354 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4217181344768 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23289872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚267054173250048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34905236 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚336887931086492 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚336887574123810 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34218604 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4209932295372 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44437893 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34218604 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23289870 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11615364 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚271401703478272 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4209592372050 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78656497 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44437893 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚267054197674496 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚332677981751760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚336198725241344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4217522130944 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚340415906586112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚340416264890880 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44427750 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44427750 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259()™fò%‘äO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚340415906586112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚340416264890880 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44427750 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44427750 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚340416264890880 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚336198742759936 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚271401861072896 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34218602 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚4399 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 % _timestamp‚1.7844887666477158e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4217522130944 _step‚354 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4217181344768 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23289872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚267054173250048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34905236 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚336887931086492 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚336887574123810 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34218604 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4209932295372 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44437893 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34218604 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23289870 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11615364 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚271401703478272 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4209592372050 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78656497 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44437893 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚267054197674496 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚332677981751760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚336198725241344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4217~Ìmø522130944 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44447782 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34912587 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚267113074007040 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78673888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44437625 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78673888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34226106 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4348619433984 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4210520595286 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4348488368128 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚340491386578432 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4218111256064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11617636 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚336961928429526 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34226104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚355 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚336272934529024 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286òŸ·‹ô&”îM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44447782 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34912587 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚267113074007040 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78673888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44437625 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78673888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34226106 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4348619433984 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4210520595286 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4348488368128 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚340491386578432 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4218111256064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11617636 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚336961928429526 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34226104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚355 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚336272934529024 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚336962285392320 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78663729 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34226106 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚332751424873600 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚340491028266496 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11740795 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11741083 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4256035437058 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚340605678275500 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23543922 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4263709166592 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34593702 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34593704 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35284715 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_ponjÍš” ol.allocated‚ 4395606637056 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34593702 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚269997989877248 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚344172022862336 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚339908331214336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚336349285875760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44923352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚274393620938752 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚340605321312818öÄÄFê'¡äO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35285005 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4264049952768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚344172022862336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚336349302915120 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚344172381167104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚339908331214336 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚339908313695744 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44933495 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34593704 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 % _timestamp‚1.7844887913934412e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79517054 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚344172381167104 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4263709166592 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79527199 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23543920 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚339908313695744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4264049952768 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11740795 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11741083 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4256035437058 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚340605678275500 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23543922 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4263709166592 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34593702 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34593704 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35284715 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4395606637056 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34593702 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚269997989877248 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚344172022862336 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚339908331214336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚336349285875760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44923352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚274393620938752 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚340605321312818 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44933495 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4256375360380æ±™é²2¢- _wandbruntime‚2268  _runtime‚2268º+ŽKî&£èM $ _timestamp‚1.784488791756639e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44933227 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79534431 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4396562070528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79544590 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4264639077888 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34601204 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚270056866209792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35292554 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35292270 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚2268.331828643 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚344247502854656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44933227 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 _step‚359 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚270056890634240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4264979871232 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚344247144542720 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4257303583728 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4256963660294 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚336422711958240 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79544590 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4264639077888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34601206 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚274453299311616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4396433101824 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44943384 trainer/global_step‚4449 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44943384 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23549001 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚340679675618534 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23549003 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚340680032581328 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚336422728997600 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934-#"È?&¤íM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚344247144542720 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4257303583728 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4256963660294 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚336422711958240 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79544590 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4264639077888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34601206 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚274453299311616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4396433101824 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44943384 trainer/global_step‚4449 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44943384 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23549001 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚340679675618534 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23549003 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚340680032581328 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚336422728997600 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 $ _timestamp‚1.784488791756639e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44933227 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79534431 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4396562070528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79544590 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4264639077888 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34601204 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚270056866209792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35292554 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35292270 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚2268.331828643 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚344247502854656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44933227 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 _step‚359 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚270056890634240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.alloc?ŸbÍ´ated‚ 4264979871232 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259çJXƒ¨2¥- _wandbruntime‚2268  _runtime‚2268ñ£I¦ 2loss/SRqqt1/assignment_loss‚0.6508006315207977 2loss/FBt2/assignment_loss‚0.011606878973485177 1loss/SRqqt2/detection_loss‚1.0774716154021555 /loss/FBt2/detection_loss‚1.3532914380942072 epoch‚5 2loss/SRqqt2/assignment_loss‚0.6809022701377279 1loss/FBt1/assignment_loss‚0.04479205485309458 /loss/FBt1/detection_loss‚0.6325765521436956 %loss/total_loss‚10.02287293345129 _step‚360 .loss/FRt1/detection_loss‚1.378797935285709 trainer/global_step‚4449 _runtime‚2268.332353964 0loss/FRt1/assignment_loss‚0.8724118782414331 0loss/FRt2/assignment_loss‚2.1148069859819207 /loss/FRt2/detection_loss‚0.5901336362924299 1loss/SRqqt1/detection_loss‚0.6152810565246319 % _timestamp‚1.7844887917575662e+09°íö §† epoch‚5 2loss/SRqqt2/assignment_loss‚0.6809022701377279 1loss/FBt1/assignment_loss‚0.04479205485309458 /loss/FBt1/detection_loss‚0.6325765521436956 %loss/total_loss‚10.02287293345129 _step‚360 .loss/FRt1/detection_loss‚1.378797935285709 trainer/global_step‚4449 _runtime‚2268.332353964 0loss/FRt1/assignment_loss‚0.8724118782414331 0loss/FRt2/assignment_loss‚2.1148069859819207 /loss/FRt2/detection_loss‚0.5901336362924299 1loss/SRqqt1/detection_loss‚0.6152810565246319 % _timestamp‚1.7844887917575662e+09 2loss/SRqqt1/assignment_loss‚0.6508006315207977 2loss/FBt2/assignment_loss‚0.011606878973485177 1loss/SRqqt2/detection_loss‚1.0774716154021555 /loss/FBt2/detection_loss‚1.3532914380942072èooWU¨‚(:Ê ãÆôÒ²áÝË# proc.memory.rssMB‚ 2783.65234375 network.sent‚ 5.946352e+06) disk./.usagePercent‚76.66454405257376 disk.nvme0c0n1.in‚ 4.3359375( disk.nvme0c0n1.out‚20662.36279296875 disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚20662.36279296875( proc.memory.availableMB‚ 562.55078125 cpu‚26.074616038850994 disk.rbd0.in‚0 disk.dm-0.in‚ 4.3359375$ memory_percent‚96.56646251678467* proc.memory.percent‚16.990065574645996 proc.cpu.threads‚17 network.recv‚636682 disk.rbd0.out‚0 disk.md0.in‚0 disk.dm-0.out‚23488.30078125" disk.dm-1.out‚3564.07470703125% disk./.usageGB‚1176.9924125671387 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 6525.83203125 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 6525.83203125 disk.md0.out‚ 63.41015625 disk.dm-1.in‚0ŒÕO“©‚(:ˆ ãÆôÒ×äÎÌ gpu.0.gpu‚58.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚54.0 gpu.0.powerWatts‚239.712' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚79.904 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n.Co2ª- _wandbruntime‚2292  _runtime‚2292b±&†Ç«Á ' lr-AdamW/pg1‚0.00020822102471699204 ' lr-AdamW/pg2‚0.00020822102471699204 trainer/global_step‚4499 $ _timestamp‚1.784488816395795e+09 _runtime‚ 2292.97077885 _step‚361»ó¦2Ì¬Æ ' lr-AdamW/pg1‚0.00020822102471699204 ' lr-AdamW/pg2‚0.00020822102471699204 trainer/global_step‚4499 $ _timestamp‚1.784488816395795e+09 _runtime‚ 2292.97077885 _step‚361é•ï)u2­- _wandbruntime‚2292  _runtime‚2292V¿Sä'®ÞO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚343617919668736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34968804 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35665343 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 $ _timestamp‚1.784488816396855e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45429097 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4310236988416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11867371 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚344323068501826 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚277385380080128 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80387756 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34968804 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚272941830928896 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34968802 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚347928139138560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚301 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45418954 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11867072 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚344323425464508 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚340020607039120 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4310236988416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚4499 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23797972 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23797970 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚347928497443328 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80397901 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45429097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4310577774592 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚299 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35665042 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4302818425388 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45418954 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚343617902150144 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚340020589999760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚343617902150144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚277385222485504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚347928139138560 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4310577774592 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚272941806504448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4310236988416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚4499 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23797972 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23797970 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚347928497443328 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80397901 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45429097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4310577774592 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚299 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35665042 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4302818425388 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45418954 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚343617902150144 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚340020589999760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚343617902150144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚277385222485504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚347928139138560 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4310577774592 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚272941806504448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34976304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚277445211248128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4444503986688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 _step‚363 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34976304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35672701 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 trainer/global_step‚4499 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34976306 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80405133 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11869932 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚348003619130880 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45428829 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4303746648736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45428829 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4311166899712 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4303406725302 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4311166899712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚344397422807542 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80415292 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚277445053660672 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35672985 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23803053 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23803051 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0]2Ìbô&²îM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4311166899712 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4303406725302 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4311166899712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚344397422807542 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80415292 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚277445053660672 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35672985 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23803053 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23803051 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚348003619130880 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚348003260818944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚343692111437824 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚340094016082240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚343692093919232 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34976304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚277445211248128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4444503986688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 _step‚363 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34976304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35672701 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 trainer/global_step‚4499 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34976306 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80405133 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11869932 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚348003619130880 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45428829 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4303746648736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45428829 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4357105596416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚280377023274496 % _timestamp‚1.7844888414773397e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚351684255414784 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35343904 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36046144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚347327508123136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚275885647556096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚275885623131648 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35343904 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11994122 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚343691911163120 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24052020 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088¸8Ï…¿äO [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚343691894123760 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚351684613719552 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81268603 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚348041172653516 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24052022 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚351684613719552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚280377180869120 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4357105596416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚280377023274496 % _timestamp‚1.7844888414773397e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚351684255414784 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35343904 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36046144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚347327508123136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚275885647556096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚275885623131648 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35343904 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11994122 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚343691911163120 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24052020 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4356764810240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4357105596416 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚347327490604544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4349261490396 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4356764810240 trainer/global_step‚4549 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81258458 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 _runtime‚2318.055439637 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2î2¢2À- _wandbruntime‚2318  _runtime‚2318?A;ï&ÁéM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚343765337245600 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚351759735407104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚351759735407104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24057103 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚275944499464192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚343765320206240 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4358035514880 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36053451 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚348115526959344 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24057101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚280437012950528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35351404 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81285994 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4492360093184 trainer/global_step‚4549 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚81275835 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚347401699892224 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81275835 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚347401682373632 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4357694721536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36053737 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚351759377095168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚347401682373632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11996350 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35351406 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4358035514880 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚348115169996550 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4350189713744 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 _runtime‚2318.426640047 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚275944523888640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45924431 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81285994 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24057101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚280437012950528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35351404 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81285994 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4492360093184 trainer/global_step‚4549 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚81275835 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚347401699892224 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81275835 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚347401682373632 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4357694721536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36053737 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚351759377095168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚347401682373632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11996350 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35351406 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4358035514880 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚348115169996550 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4350189713744 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 _runtime‚2318.426640047 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚275944523888640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45924431 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81285994 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚351759735407104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24057103 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚275944499464192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚343765320206240 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4358035514880 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36053451 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592ïFmŸè2Ã- _wandbruntime‚2318  _runtime‚2318¬ý:èÄÿ /loss/FBt1/detection_loss‚0.7452640090706437 /loss/FBt2/detection_loss‚0.6430818803076234 /loss/FRt1/assignment_loss‚1.135769460849835 1loss/SRqqt2/detection_loss‚0.9923365981370773 epoch‚5 % _timestamp‚1.7844888418525987e+09 _runtime‚2318.430407934 /loss/FRt2/assignment_loss‚2.110835511998816 2loss/SRqqt2/assignment_loss‚0.6269560128907269 1loss/FBt1/assignment_loss‚0.06997493305390634 3loss/FBt2/assignment_loss‚0.0010370403384172278 %loss/total_loss‚9.492496804945663 trainer/global_step‚4549 0loss/SRqqt1/assignment_loss‚0.59555934961868 /loss/FRt2/detection_loss‚0.5554311023661125 1loss/SRqqt1/detection_loss‚0.5859898865537744 _step‚368 /loss/FRt1/detection_loss‚1.4302610197600505ŒVA‰ Å„ 1loss/SRqqt2/detection_loss‚0.9923365981370773 epoch‚5 % _timestamp‚1.7844888418525987e+09 _runtime‚2318.430407934 /loss/FRt2/assignment_loss‚2.110835511998816 2loss/SRqqt2/assignment_loss‚0.6269560128907269 1loss/FBt1/assignment_loss‚0.06997493305390634 3loss/FBt2/assignment_loss‚0.0010370403384172278 %loss/total_loss‚9.492496804945663 trainer/global_step‚4549 0loss/SRqqt1/assignment_loss‚0.59555934961868 /loss/FRt2/detection_loss‚0.5554311023661125 1loss/SRqqt1/detection_loss‚0.5859898865537744 _step‚368 /loss/FRt1/detection_loss‚1.4302610197600505 /loss/FBt1/detection_loss‚0.7452640090706437 /loss/FBt2/detection_loss‚0.6430818803076234 /loss/FRt1/assignment_loss‚1.135769460849835ð^ÚhÍOÆ‚(:Ä ÇôÒ»«ÃË% disk.nvme1c1n1.out‚6534.361328125 proc.cpu.threads‚17# disk.nvme1n1.out‚6534.361328125 disk.md0.out‚ 64.453125 disk.dm-0.in‚ 4.3359375 disk.rbd0.out‚0' disk.nvme0c0n1.out‚20668.1728515625$ memory_percent‚96.56696319580078% proc.memory.availableMB‚ 562.46875# proc.memory.rssMB‚ 2783.65234375( disk./.usagePercent‚76.6645828145196$ disk./.usageGB‚1176.993007659912 disk.nvme1n1.in‚0" disk.dm-1.out‚3564.40185546875% disk.nvme0n1.out‚20668.1728515625* proc.memory.percent‚16.990065574645996 cpu‚26.0722231998758 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 network.sent‚ 6.075953e+06 network.recv‚648469 disk.md0.in‚0# disk.dm-0.out‚23500.04150390625 disk.nvme1c1n1.in‚0%kQfžÇ‚(:“ ÇôÒ§žµË gpu.0.gpu‚66.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚52.0 gpu.0.powerWatts‚130.968' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚43.65599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ²0XÈ‚(:Í ŸÇôÒ¥ŽæË proc.cpu.threads‚17 disk.md0.in‚0' disk.nvme0c0n1.out‚20669.0537109375 disk.nvme0n1.in‚ 4.3359375$ memory_percent‚96.56548500061035) disk./.usagePercent‚76.66459250500606% disk./.usageGB‚1176.9931564331055 disk.md0.out‚ 64.8046875 disk.dm-0.in‚ 4.3359375 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0' proc.memory.availableMB‚ 562.7109375 network.sent‚ 6.13905e+06# disk.dm-0.out‚23500.61181640625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.3359375% disk.nvme0n1.out‚20669.0537109375' disk.nvme1c1n1.out‚6535.07763671875% disk.nvme1n1.out‚6535.07763671875# proc.memory.rssMB‚ 2783.65234375 network.recv‚651596 disk.dm-1.in‚0! disk.dm-1.out‚3564.6669921875* proc.memory.percent‚16.990065574645996 cpu‚26.070057404767148þèýÉ‚(:’ ŸÇôÒåÿ£Ì gpu.0.gpu‚94.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚52.0 gpu.0.powerWatts‚117.8' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.266666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛÔ“2Ê- _wandbruntime‚2343  _runtime‚2343Áƒ+ÑÇËÁ % _timestamp‚1.7844888665173914e+09 _runtime‚2343.092373988 _step‚369 & lr-AdamW/pg1‚0.0002079692865522552 & lr-AdamW/pg2‚0.0002079692865522552 trainer/global_step‚4599BM¯<ÌÌÆ trainer/global_step‚4599 % _timestamp‚1.7844888665173914e+09 _runtime‚2343.092373988 _step‚369 & lr-AdamW/pg1‚0.0002079692865522552 & lr-AdamW/pg2‚0.0002079692865522552ñ ß¾‰2Í- _wandbruntime‚2343  _runtime‚2343.cxDüÎßO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12120070 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚355440371691008 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚351037096577536 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36426424 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚4599 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚278829464183296 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4403292632064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7844888665183582e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82139305 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4539502066688 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35719002 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚351037079058944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4539364702208 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚355440729995776 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24306072 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 137364480 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚351758562879842 _runtime‚2343.093274849 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35719004 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4403292632064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_spú?¬hélit_bytes.all.allocated‚283368966249984 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚355440371691008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12120352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4395704555404 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161788928 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚351758919842524 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46420301 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4403633418240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4403633418240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36426140 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82129160 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46420301 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚347363198247760 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446èTÆ•ê'ÏäO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4403292632064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7844888665183582e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82139305 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4539502066688 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35719002 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚351037079058944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4539364702208 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚355440729995776 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24306072 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 137364480 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚351758562879842 _runtime‚2343.093274849 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35719004 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4403292632064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚283368966249984 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚355440371691008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12120352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4395704555404 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161788928 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚351758919842524 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46420301 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4403633418240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4403633418240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36426140 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82129160 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46420301 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚347363198247760 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12120070 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚355440371691008 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚351037096577536 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36426424 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚4599 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚278829464183296òß÷Mo2Ð- _wandbruntime‚2343  _runtime‚2343’-\Æï&ÑéM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46430190 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36434020 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36433737 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚351832917185558 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24311151 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚278888340515840 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚347436641369600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24311153 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46430190 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82156696 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚283428800965120 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35726504 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚351111288346624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚351111270828032 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4540460449280 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35726506 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46420033 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12122586 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚278888316091392 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4540320994816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚351833274148352 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4396292855318 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4404222543360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚4599 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚283428637086208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 _runtime‚2343.464007837 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35726506 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4396632778752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4404563336704qÉéÒîM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚351832917185558 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24311151 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚278888340515840 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚347436641369600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24311153 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46430190 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82156696 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚283428800965120 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35726504 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚351111288346624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚351111270828032 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4540460449280 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35726506 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46420033 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12122586 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚278888316091392 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4540320994816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚351833274148352 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4396292855318 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4404222543360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.a¡i ll.freed‚0 trainer/global_step‚4599 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚283428637086208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 _runtime‚2343.464007837 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35726506 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4396632778752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4404563336704 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46430190 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36434020 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36433737 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088ó«€Ô%2Ó- _wandbruntime‚2343  _runtime‚2343©?ÑÔ€ 0loss/FRt2/detection_loss‚0.49724298483812956 trainer/global_step‚4599 2loss/SRqqt2/assignment_loss‚0.7168308134690354 3loss/FBt2/assignment_loss‚0.0018796568160570509 _runtime‚2343.464602028 _step‚372 2loss/SRqqt1/assignment_loss‚0.5890286322036639 1loss/SRqqt1/detection_loss‚0.6246425965941991 /loss/FBt1/detection_loss‚0.6071188634687132 .loss/FBt2/detection_loss‚0.715264508990865 "loss/total_loss‚9.536982554723 % _timestamp‚1.7844888668899298e+09 /loss/FRt2/assignment_loss‚2.260940206983377 /loss/FRt1/detection_loss‚1.3570647446943267 1loss/SRqqt2/detection_loss‚1.1387477679527365 epoch‚5 0loss/FRt1/assignment_loss‚1.0012227946699264 2loss/FBt1/assignment_loss‚0.026998984041970705šABú Õ… trainer/global_step‚4599 2loss/SRqqt2/assignment_loss‚0.7168308134690354 3loss/FBt2/assignment_loss‚0.0018796568160570509 _runtime‚2343.464602028 _step‚372 2loss/SRqqt1/assignment_loss‚0.5890286322036639 1loss/SRqqt1/detection_loss‚0.6246425965941991 /loss/FBt1/detection_loss‚0.6071188634687132 .loss/FBt2/detection_loss‚0.715264508990865 "loss/total_loss‚9.536982554723 % _timestamp‚1.7844888668899298e+09 /loss/FRt2/assignment_loss‚2.260940206983377 /loss/FRt1/detection_loss‚1.3570647446943267 1loss/SRqqt2/detection_loss‚1.1387477679527365 epoch‚5 0loss/FRt1/assignment_loss‚1.0012227946699264 2loss/FBt1/assignment_loss‚0.026998984041970705 0loss/FRt2/detection_loss‚0.49724298483812956ô~Mœ[VÖ‚(:Ë ®ÇôÒ‘ø³Ë proc.cpu.threads‚17 disk.nvme1c1n1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚ 6.143107e+06' disk.nvme1c1n1.out‚6535.77099609375% disk.nvme1n1.out‚6535.77099609375 disk.nvme0n1.in‚ 4.3359375 proc.memory.rssMB‚ 2783.65625 cpu‚26.06793429371136 network.recv‚655000% disk./.usageGB‚1176.7555389404297 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚23502.43798828125! disk.dm-1.out‚3564.8154296875 disk.nvme0c0n1.in‚ 4.3359375% disk.nvme0n1.out‚20670.8779296875$ memory_percent‚96.56589031219482( disk./.usagePercent‚76.6491150588132 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 65.05078125' disk.nvme0c0n1.out‚20670.8779296875( proc.memory.availableMB‚ 562.64453125* proc.memory.percent‚16.990089416503906'Œþž×‚(:“ ®ÇôÒÂç’Ì gpu.0.gpu‚55.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚52.0 gpu.0.powerWatts‚118.024' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.34133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0λø½2Ø- _wandbruntime‚2368  _runtime‚2368ݾ=qÆÙÀ & lr-AdamW/pg2‚0.0002078409767896509 trainer/global_step‚4649 $ _timestamp‚1.784488891571201e+09 _runtime‚2368.146186266 _step‚373 & lr-AdamW/pg1‚0.0002078409767896509‡æ^NËÚÅ & lr-AdamW/pg1‚0.0002078409767896509 & lr-AdamW/pg2‚0.0002078409767896509 trainer/global_step‚4649 $ _timestamp‚1.784488891571201e+09 _runtime‚2368.146186266 _step‚373õºÌa÷2Û- _wandbruntime‚2368  _runtime‚2368&9Š¤å'ÜßO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚351034519411120 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚354746667513344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46905760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _runtime‚2368.150454603 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36805983 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚359196846272000 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36094102 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4442147620412 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46915903 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚354746685031936 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46905760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4449820453888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83010007 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36094104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24560120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4450161240064 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36094102 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12245863 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚359196846272000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚286360702081536 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚281773280810496 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36806274 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24560122 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46915903 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚359196487967232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82999862 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4587292295168 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4449820453888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚281773256386048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4441807697090 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4450161240064 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚286360548681216 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚351034502371760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12246152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚354746667513344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0Ç/xjê'ÝäO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36094102 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12245863 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚359196846272000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚286360702081536 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚281773280810496 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36806274 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24560122 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46915903 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚359196487967232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82999862 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4587292295168 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4449820453888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚281773256386048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4441807697090 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4450161240064 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚286360548681216 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚351034502371760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12246152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚354746667513344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚351034519411120 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚354746667513344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46905760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _runtime‚2368.150454603 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36805983 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚359196846272000 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36094102 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4442147620412 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46915903 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚354746685031936 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46905760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4449820453888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83010007 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36094104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24560120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4450161240064 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834ö&UÊ(2Þ- _wandbruntime‚2368  _runtime‚2368H ºõfßéM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46915635 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚351107945493600 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83017239 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4451091158528 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 trainer/global_step‚4649 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚351107928454240 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4443075843760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚354820859282432 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚355551021337360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚355550664374566 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83017239 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36101604 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46925792 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚281832157143040 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚354820859282432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83027398 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46925792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4588379964928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚354820876801024 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4450750365184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24565201 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 % _timestamp‚1.7844888919456983e+09 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83027398 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36101606 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚359271609647616 _step‚375 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚354820876801024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 _runtime‚2368.520811213 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚286420537107968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4588246801920 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4451091158528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚281832132718592 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36101606 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46915635 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12248662 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36101604 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚359271967959552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36813575 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088¾æ!ô&àîM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4451091158528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚281832132718592 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36101606 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46915635 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12248662 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36101604 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚359271967959552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36813575 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46915635 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚351107945493600 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83017239 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4451091158528 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 trainer/global_step‚4649 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚351107928454240 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4443075843760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚354820859282432 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚355551021337360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚355550664374566 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83017239 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36101604 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46925792 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚281832157143040 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚354820859282432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83027398 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46925792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4588379964928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚354820876801024 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4450750365184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24565201 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 % _timestamp‚1.7844888919456983e+09 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83027398 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36101606 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚359271609647616 _step‚375 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚354820876801024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 _runtime‚2368.520811213 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚286420537107968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4588246801920 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720÷›öá2á- _wandbruntime‚2368  _runtime‚2368åHâ 0loss/FRt2/assignment_loss‚2.0867807143102817 1loss/SRqqt1/detection_loss‚0.6554847197136003 1loss/SRqqt2/detection_loss‚0.9594060844754347 0loss/FRt1/assignment_loss‚0.9713086453009778 2loss/SRqqt2/assignment_loss‚0.6213865981333436 /loss/FRt1/detection_loss‚1.3619863251004163 /loss/FRt2/detection_loss‚0.5261668528906236 /loss/FBt1/detection_loss‚0.4676968396905291 /loss/FBt2/detection_loss‚1.3603211048757657 %loss/total_loss‚9.640670867677933 epoch‚5 1loss/FBt1/assignment_loss‚0.06989159679938345 trainer/global_step‚4649 % _timestamp‚1.7844888919467173e+09 _runtime‚2368.524406599 _step‚376 2loss/SRqqt1/assignment_loss‚0.5376889032087149 1loss/FBt2/assignment_loss‚0.02255248317886327$u˜× ㆠ2loss/SRqqt2/assignment_loss‚0.6213865981333436 /loss/FRt1/detection_loss‚1.3619863251004163 /loss/FRt2/detection_loss‚0.5261668528906236 /loss/FBt1/detection_loss‚0.4676968396905291 /loss/FBt2/detection_loss‚1.3603211048757657 %loss/total_loss‚9.640670867677933 epoch‚5 1loss/FBt1/assignment_loss‚0.06989159679938345 trainer/global_step‚4649 % _timestamp‚1.7844888919467173e+09 _runtime‚2368.524406599 _step‚376 2loss/SRqqt1/assignment_loss‚0.5376889032087149 1loss/FBt2/assignment_loss‚0.02255248317886327 0loss/FRt2/assignment_loss‚2.0867807143102817 1loss/SRqqt1/detection_loss‚0.6554847197136003 1loss/SRqqt2/detection_loss‚0.9594060844754347 0loss/FRt1/assignment_loss‚0.9713086453009778øœbÄ•Tä‚(:É ½ÇôÒôËŠÌ network.sent‚ 6.218164e+06# disk.dm-0.out‚23503.16455078125 disk.rbd0.in‚0% disk.nvme0c0n1.out‚20671.98046875 disk.nvme0n1.in‚ 4.3359375# disk.nvme0n1.out‚20671.98046875$ memory_percent‚96.56760692596436 proc.memory.rssMB‚ 2783.65625) disk./.usagePercent‚76.64911232559908 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.3359375' disk.nvme1c1n1.out‚6536.46923828125* proc.memory.percent‚16.990089416503906 proc.cpu.threads‚17 network.recv‚659147 disk.md0.out‚ 65.48828125! disk.dm-1.out‚3564.9345703125% disk.nvme1n1.out‚6536.46923828125( proc.memory.availableMB‚ 562.36328125 cpu‚26.065813414316423% disk./.usageGB‚1176.7554969787598 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0¿œå‚(:‘ ½ÇôÒ«¨½Ì gpu.0.gpu‚7.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚52.0 gpu.0.powerWatts‚118.135' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.37833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09e‘ Tæ‚(:É ÌÇôÒ’ñÅË network.recv‚663321) disk./.usagePercent‚76.63367935625432# disk.nvme1n1.out‚6537.212890625 disk.md0.in‚0 disk.rbd0.out‚0.0625 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375% disk./.usageGB‚1176.5185623168945 disk.nvme1n1.in‚0 disk.dm-1.in‚0! disk.dm-1.out‚3565.0556640625 disk.rbd0.in‚0 cpu‚26.06468459128011 proc.cpu.threads‚17 network.sent‚ 6.266495e+06 disk.dm-0.in‚ 4.3359375" disk.dm-0.out‚23510.3486328125& disk.nvme0c0n1.out‚20679.349609375$ memory_percent‚96.56774997711182( proc.memory.availableMB‚ 562.33984375* proc.memory.percent‚16.990089416503906 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚6537.212890625 disk.md0.out‚ 65.8671875$ disk.nvme0n1.out‚20679.349609375 proc.memory.rssMB‚ 2783.65625ÿi˜pžç‚(:“ ÌÇôÒªÌÃË gpu.0.gpu‚92.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚250.789' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚83.59633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0d¡2è- _wandbruntime‚2393  _runtime‚2393%gOÇéÁ % _timestamp‚1.7844889166515772e+09 _runtime‚2393.226556628 _step‚377 & lr-AdamW/pg1‚0.0002077110426383941 & lr-AdamW/pg2‚0.0002077110426383941 trainer/global_step‚4699˜ÏË!ÌêÆ & lr-AdamW/pg2‚0.0002077110426383941 trainer/global_step‚4699 % _timestamp‚1.7844889166515772e+09 _runtime‚2393.226556628 _step‚377 & lr-AdamW/pg1‚0.0002077110426383941ùmýë2ë- _wandbruntime‚2393  _runtime‚2393™˜†å'ìßO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47401362 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚362952604243456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12371580 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4488250762098 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚354705806495760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚289352482219008 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚354705823535120 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 _runtime‚2393.227533859 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36469202 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47411505 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4635257902592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚359194057257858 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4488590685420 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 % _timestamp‚1.7844889166525664e+09 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4635384781312 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36469204 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚362952604243456 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4496348275712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4496689061888 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37185750 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚358456255967744 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚358456273486336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83870564 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12371863 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37186035 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24814170 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47411505 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚362952962548224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚358456255967744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4496348275712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 _step‚378 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚358456273486336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 trainer/global_step‚4699 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83880709 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36469204 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚289352330915840 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚284717073013248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36469202[Ûœ- íäO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚358456273486336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 trainer/global_step‚4699 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83880709 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36469204 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚289352330915840 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚284717073013248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36469202 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47401362 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚362952604243456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12371580 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4488250762098 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚354705806495760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚289352482219008 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚354705823535120 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 _runtime‚2393.227533859 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36469202 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47411505 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4635257902592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚359194057257858 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4488590685420 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 % _timestamp‚1.7844889166525664e+09 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4635384781312 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36469204 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚362952604243456 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4496348275712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4496689061888 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37185750 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚358456255967744 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚358456273486336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83870564 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12371863 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37186035 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24814170 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47411505 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚362952962548224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚358456255967744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4496348275712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 _step‚378úñæV42î- _wandbruntime‚2393  _runtime‚2393Þéâ‚ï&ïéM EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2393.606689114 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37193410 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚358530447736832 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4489178985334 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚358530447736832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83887941 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36476706 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83898100 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83887941 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚363027725923840 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36476706 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83898100 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37193685 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚289412169037824 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚284775973770240 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4636342369280 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12374159 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚359268768526368 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚358530465255424 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚379 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚289412316139520 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚354779232578240 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4497278187008 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚358530465255424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4489518908768 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24819251 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7844889170315232e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47421394 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47411237 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2393.606689114 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37193410 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚358530447736832 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4489178985334 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚358530447736832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83887941 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36476706 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83898100 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83887941 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚363027725923840 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36476706 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83898100 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37193685 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚289412169037824 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚284775973770240 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4636342369280 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12374159 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚359268768526368 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚358530465255424 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚379 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚289412316139520 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚354779232578240 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4497278187008 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚358530465255424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4489518908768 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24819251 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7844889170315232e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47421394 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47411237 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 543162368 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 # _timestamp‚1.78448894173767e+09 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36844304 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚362165861940¬‚× O#736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36844304 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47907107 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚362912161409548 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚358377127659120 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚287660914064896 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47896964 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12498172 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚382 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36844302 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚366708720519680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12497885 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4683412501504 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4543216883712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47907107 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚366709078824448 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4542876097536 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚292344173166080 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25068222 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚4749 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚362165844422144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36844302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4535033750428 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366708720519680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25068220 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚362911804446866 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84741266 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4542876097536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚287660889640448 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚362165861940736 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4543216883712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚366709078824448 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚358377110619760 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚292344326566400èØÍè'ýâO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚259 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4543216883712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚366709078824448 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚358377110619760 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 543162368 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚292344326566400 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84751411 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4534693827106 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚362165844422144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 # _timestamp‚1.78448894173767e+09 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36844304 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚362165861940736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36844304 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47907107 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚362912161409548 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚358377127659120 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚287660914064896 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 543162368 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47896964 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12498172 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚259 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚382 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36844302 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚366708720519680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12497885 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4683412501504 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4543216883712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚934 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47907107 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚366709078824448 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4542876097536 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚292344173166080 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 543162368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14405337088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25068222 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚4749 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚362165844422144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36844302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4535033750428 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366708720519680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25068220 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚362911804446866 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84741266 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4542876097536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚287660889640448 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14405337088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚362165861940736 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚259þÇÓô„2þ- _wandbruntime‚2418  _runtime‚24188:Kùî&ÿèM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47916996 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47906839 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36851806 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚292404159335936 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚358450553741600 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4543806008832 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37573989 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 % _timestamp‚1.7844889421095903e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚287719790397440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚362240053709824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚362240036191232 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4684368938496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84768802 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12500686 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚362986515715376 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 _step‚383 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚362240053709824 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 _runtime‚ 2418.68472869 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47906839 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4544146802176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚287719765972992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4544146802176 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37573701 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚366784200512000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84768802 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84758643 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4535622050342 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4749 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36851806 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4543806008832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25073301 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚366784200512000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84758643 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚362240036191232 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4684368938496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171467264 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14405337088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84768802 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12500686 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚362986515715376 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚259 _step‚383 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚362240053709824 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 _runtime‚ 2418.68472869 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47906839 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4544146802176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚287719765972992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4544146802176 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37573701 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚366784200512000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 )dðómM=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84768802 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84758643 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4535622050342 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4749 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36851806 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4543806008832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25073301 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚934 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚366784200512000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84758643 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4543806008832 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37573989 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 % _timestamp‚1.7844889421095903e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚287719790397440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368ÿ$A³©2- _wandbruntime‚2418  _runtime‚2418ÑÞÙ'‚þ /loss/FBt1/detection_loss‚0.6531923441355353 0loss/FRt2/assignment_loss‚2.1788670544103947 -loss/FRt1/detection_loss‚1.40336043087811 1loss/SRqqt1/detection_loss‚0.6010566325090805 trainer/global_step‚4749 % _timestamp‚1.7844889421106465e+09 2loss/FBt1/assignment_loss‚0.008546049538091502 /loss/FRt2/detection_loss‚0.5116933207089355 epoch‚5 /loss/FBt2/detection_loss‚0.6196174191212968 /loss/FRt1/assignment_loss‚0.898784548563409 1loss/SRqqt1/assignment_loss‚0.661405104715195 2loss/SRqqt2/assignment_loss‚0.4869920116435804 2loss/FBt2/assignment_loss‚0.005022375572498224 0loss/SRqqt2/detection_loss‚1.073585083014629 %loss/total_loss‚9.102122374810756 _runtime‚2418.685313791 _step‚384اW ƒƒ 1loss/SRqqt1/assignment_loss‚0.661405104715195 2loss/SRqqt2/assignment_loss‚0.4869920116435804 2loss/FBt2/assignment_loss‚0.005022375572498224 0loss/SRqqt2/detection_loss‚1.073585083014629 %loss/total_loss‚9.102122374810756 _runtime‚2418.685313791 _step‚384 /loss/FBt1/detection_loss‚0.6531923441355353 0loss/FRt2/assignment_loss‚2.1788670544103947 -loss/FRt1/detection_loss‚1.40336043087811 1loss/SRqqt1/detection_loss‚0.6010566325090805 trainer/global_step‚4749 % _timestamp‚1.7844889421106465e+09 2loss/FBt1/assignment_loss‚0.008546049538091502 /loss/FRt2/detection_loss‚0.5116933207089355 epoch‚5 /loss/FBt2/detection_loss‚0.6196174191212968 /loss/FRt1/assignment_loss‚0.898784548563409€^ì>9Z„‚(:Ï ùÇôÒ Ø€Ì disk.nvme0n1.in‚ 4.3359375" proc.memory.rssMB‚ 2783.6640625 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0# disk.nvme1n1.out‚6547.818359375 disk.md0.out‚ 66.6484375" disk.dm-0.out‚23545.0673828125" disk.dm-1.out‚3565.43408203125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375$ memory_percent‚96.56696319580078% proc.memory.availableMB‚ 562.46875* proc.memory.percent‚16.990137100219727 cpu‚26.062664918150254 proc.cpu.threads‚17 network.recv‚674143) disk./.usagePercent‚76.64083292305305% disk.nvme1c1n1.out‚6547.818359375 disk.md0.in‚0 disk.rbd0.out‚0.34375( disk.nvme0c0n1.out‚20705.57666015625& disk.nvme0n1.out‚20705.57666015625 network.sent‚ 6.351398e+06% disk./.usageGB‚1176.6283874511719 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0SÌ6K…‚(:’ ùÇôÒ…æ¶Ì gpu.0.gpu‚71.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚54.0 gpu.0.powerWatts‚246.61' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚82.20333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸºMô2†- _wandbruntime‚2442  _runtime‚2442êŽ*áLJÁ _step‚385 & lr-AdamW/pg1‚0.0002074463095205443 & lr-AdamW/pg2‚0.0002074463095205443 trainer/global_step‚4799 % _timestamp‚1.7844889662347496e+09 _runtime‚2442.809840028'Ó^“ÌˆÆ & lr-AdamW/pg2‚0.0002074463095205443 trainer/global_step‚4799 % _timestamp‚1.7844889662347496e+09 _runtime‚2442.809840028 _step‚385 & lr-AdamW/pg1‚0.0002074463095205443zÀO2‰- _wandbruntime‚2442  _runtime‚2442Y‘lå'ŠßO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4589403919360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37219402 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4581136892114 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37946290 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12624020 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48402709 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124781568 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚362048414743760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚934 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 _runtime‚2442.811653601 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37946573 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚370465195100672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25322272 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚934 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚934 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14405337088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚365875450395136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171467264 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4589744705536 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚370465195100672 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚370464836795904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4589744705536 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚370465195100672 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚370464836795904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446‚6„ ‚2Œ- _wandbruntime‚2443  _runtime‚2443Õj~ èM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚365949642164224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85629345 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4590674624000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚365949624645632 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85639504 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37226906 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚366704262904384 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚362121857865600 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚365949624645632 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 YGDeviceStatsMonitor.on_traiêÃè‚ën_batch_end/active_bytes.small_pool.allocated‚ 4590674624000 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48412598 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37953846 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12626495 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚387 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12626779 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4732247357440 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37226904 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚370539958476288 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4590333830656 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 trainer/global_step‚4799 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85629345 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37226906 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 _runtime‚2443.171898998 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚295395985447936 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37226904 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚362121840826240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚295395829957632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0éj¥Îó&ŽíM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37226906 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚934 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14405337088 _runtime‚2443.171898998 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚295395985447936 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14405337088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37226904 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚362121840826240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚295395829957632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚365949642164224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 543162368 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85629345 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4590674624000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚365949624645632 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85639504 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37226906 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚366704262904384 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚362121857865600 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚259 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚365949624645632 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4590674624000 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48412598 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37953846 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12626495 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚387 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12626779 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4732247357440 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 543162368 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37226904 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚934 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚370539958476288 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4590333830656 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚259 trainer/global_step‚4799 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85629345 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚934ƒBó’È2- _wandbruntime‚2443  _runtime‚2443«ˆ¤ ƒ /loss/FBt2/detection_loss‚0.7002857583848869 %loss/total_loss‚8.872723956926425 _runtime‚2443.172424389 epoch‚5 2loss/FBt2/assignment_loss‚0.006980864762930975 1loss/SRqqt2/detection_loss‚0.9448907773565645 _step‚388 0loss/FRt1/assignment_loss‚0.9910530644150737 2loss/SRqqt1/assignment_loss‚0.5560344887719773 1loss/SRqqt2/assignment_loss‚0.667573323200206 2loss/FBt1/assignment_loss‚0.023702832889562733 /loss/FRt2/detection_loss‚0.5538060655161154 1loss/SRqqt1/detection_loss‚0.6592391979151713 trainer/global_step‚4799 % _timestamp‚1.7844889665976548e+09 0loss/FRt2/assignment_loss‚2.1071719844371843 /loss/FRt1/detection_loss‚1.2288005679442917 0loss/FBt1/detection_loss‚0.43318503133246095¯žü‘ˆ trainer/global_step‚4799 % _timestamp‚1.7844889665976548e+09 0loss/FRt2/assignment_loss‚2.1071719844371843 /loss/FRt1/detection_loss‚1.2288005679442917 0loss/FBt1/detection_loss‚0.43318503133246095 /loss/FBt2/detection_loss‚0.7002857583848869 %loss/total_loss‚8.872723956926425 _runtime‚2443.172424389 epoch‚5 2loss/FBt2/assignment_loss‚0.006980864762930975 1loss/SRqqt2/detection_loss‚0.9448907773565645 _step‚388 0loss/FRt1/assignment_loss‚0.9910530644150737 2loss/SRqqt1/assignment_loss‚0.5560344887719773 1loss/SRqqt2/assignment_loss‚0.667573323200206 2loss/FBt1/assignment_loss‚0.023702832889562733 /loss/FRt2/detection_loss‚0.5538060655161154 1loss/SRqqt1/detection_loss‚0.6592391979151713„ý@E[’‚(:Ð ˆÈôÒ±¼Ë disk.rbd0.out‚ 0.40673828125& disk.nvme0c0n1.out‚21028.283203125 cpu‚26.064633009205046 proc.cpu.threads‚17 network.recv‚678251# disk.dm-0.out‚23958.72509765625" disk.dm-1.out‚3565.62158203125 disk.nvme0c0n1.in‚ 4.3359375* proc.memory.percent‚16.990137100219727 network.sent‚ 6.401003e+06% disk./.usageGB‚1176.9954452514648! disk.nvme1n1.out‚ 6640.3203125 disk.rbd0.in‚0 disk.nvme0n1.in‚ 4.3359375$ memory_percent‚96.56858444213867) disk./.usagePercent‚76.66474158941311 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375$ disk.nvme0n1.out‚21028.283203125 disk.nvme1c1n1.in‚0# disk.nvme1c1n1.out‚ 6640.3203125& proc.memory.availableMB‚ 562.203125" proc.memory.rssMB‚ 2783.6640625 disk.nvme1n1.in‚0 disk.md0.out‚ 67.30078125 disk.dm-1.in‚0k>2 ž“‚(:“ ˆÈôÒøæ¡Ì gpu.0.gpu‚63.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚54.0 gpu.0.powerWatts‚267.054' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.01799999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³Z]_”‚(:Ô —ÈôÒþÆóË disk.rbd0.out‚ 0.40673828125% disk.nvme0c0n1.out‚21065.97265625 disk.nvme0n1.in‚ 4.3359375 disk.md0.in‚0 disk.nvme1c1n1.in‚0 disk.md0.out‚ 67.55078125 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚24030.55712890625 disk.dm-1.out‚3565.775390625( proc.memory.availableMB‚ 512.44921875) proc.memory.percent‚17.29428768157959 proc.cpu.threads‚17' disk.nvme1c1n1.out‚6675.18310546875% disk.nvme1n1.out‚6675.18310546875# disk.nvme0n1.out‚21065.97265625 disk.dm-1.in‚0$ memory_percent‚96.87225818634033 cpu‚26.0840382516552% disk./.usageGB‚1176.9945449829102 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1n1.in‚0# proc.memory.rssMB‚ 2833.49609375 network.sent‚ 6.465388e+06 network.recv‚682390) disk./.usagePercent‚76.66468294954633 disk.rbd0.in‚0MËõõ’•‚(:‡ —ÈôÒÊŸ¤Ì gpu.0.gpu‚33.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.410659790039062* gpu.0.memoryAllocatedBytes‚ 14955642880 gpu.0.temp‚53.0 gpu.0.powerWatts‚251.289' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚83.763 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-6 Å2–- _wandbruntime‚2465  _runtime‚2465Û^r…p*—™{ 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6985437273979187 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012728306464850903 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.09935101866722107 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1029326319694519 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0714285746216774 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.993574857711792 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6765905618667603 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.917954683303833 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.986619770526886 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9979727864265442 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.996444046497345 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6816357970237732 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.039519548416137695 *particle/sensitivity‚0.625562846660614 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9903395771980286 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10686425119638443 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4249792993068695 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8482905626296997 jet/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9970992207527161 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.16967138648033142 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9961899518966675 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7381882667541504 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9885439872741699 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5897520184516907 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.10606767237186432 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8166688084602356 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.10141504555940628 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7validation_average_jet_accuracy‚0.32340285181999207 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.643570065498352 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6072615385055542 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.10769183188676834 *jet/accuracy_1_of_4‚0.9582488536834717 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8486841917037964 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6713894605636597 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6155487895011902 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.11793318390846252 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +particle/specificity‚0.9032500386238098 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3533458113670349 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8574560880661011 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.7535116076469421 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8504273891448975 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07727242261171341 +jet/accuracy_4_of_4‚0.01666666567325592 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.18092086911201477 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03653086721897125 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.07377177476882935 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 jet/accuracy_2_of_5‚1 jet/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.528556764125824 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10657672584056854 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5573469400405884 'particle/f_score‚0.6320435404777527 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5937500596046448 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6859825849533081 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 (particle/accuracy‚0.8436259031295776 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37654319405555725 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_4‚1 jet/accuracy_1_of_5‚1 )loss/val_total_loss‚8.894206720055362 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4626079797744751 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02767857164144516 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07500000298023224 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7619657516479492 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.879057765007019 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6145364046096802 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01194691937416792 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9967120289802551 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5949175357818604 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9504438042640686 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5441714525222778 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6827719807624817 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07500000298023224 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6183421015739441 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9770880937576294 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9916524291038513 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.08317320048809052 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_5‚0 _step‚389 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 *jet/accuracy_1_of_2‚0.7451939582824707 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9285714030265808 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1029326319694519 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6221818327903748 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12402790784835815 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.995132565498352 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6804730296134949 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4739980697631836 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_5_of_5‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16656892001628876 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 trainer/global_step‚4805 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.006570411380380392 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8952308893203735 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.622810959815979 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9856441020965576 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.11502408981323242 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.09878669679164886 particle/accuracy_1_of_1‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7799999713897705 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02359246462583542 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.11756092309951782 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3788035809993744 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03919782117009163 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8969069719314575 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988510608673096 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.11829479783773422 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6933782696723938 /particle/accuracy_3_of_4‚0.9955229759216309 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 particle/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8563199639320374 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.08342001587152481 +jet/accuracy_2_of_3‚0.47508883476257324 particle/accuracy_3_of_5‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/*FRt2SRqqt1«dâ&/FBt/event_purity‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.023766914382576942 particle/accuracy_1_of_5‚1 epoch‚5 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.941826343536377 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1313161551952362 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7799999713897705 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0714285746216774 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.43692535161972046 *jet/accuracy_1_of_1‚0.2743188142776489 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5752818584442139 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.11638996750116348 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7523455619812012 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 _runtime‚ 2465.84725494 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40909090638160706 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6413089036941528 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6405024528503418 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6137741804122925 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.20901896059513092 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6171441674232483 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6080068945884705 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7528355121612549 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6868059635162354 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6507748365402222 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03619508445262909 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6915391087532043 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 .particle/accuracy_3_of_3‚0.999514102935791 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09071813523769379 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6137741804122925 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49000000953674316 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3194444477558136 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6477194428443909 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12234789878129959 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0035892738960683346 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8355262875556946 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.545059323310852 particle/accuracy_2_of_2‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.10435999929904938 +jet/accuracy_3_of_4‚0.09441577643156052 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2511190176010132 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9957055449485779 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13124999403953552 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6839740872383118 particle/accuracy_1_of_2‚1 % _timestamp‚1.7844889892716777e+09 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8166688084602356 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28377479314804077 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6105924844741821 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.663498044013977 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02767857164144516 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3682684898376465 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3561813235282898 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6118217706680298 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.530468761920929 +jet/accuracy_2_of_2‚0.13125984370708466 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992743134498596 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.10144618898630142 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957ñO¸è¤=˜ž{ 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6145364046096802 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01194691937416792 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9967120289802551 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5949175357818604 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9504438042640686 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5441714525222778 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6827719807624817 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07500000298023224 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6183421015739441 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9770880937576294 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9916524291038513 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.08317320048809052 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_5‚0 _step‚389 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 *jet/accuracy_1_of_2‚0.7451939582824707 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9285714030265808 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1029326319694519 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6221818327903748 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12402790784835815 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.995132565498352 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6804730296134949 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4739980697631836 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_5_of_5‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16656892001628876 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 trainer/global_step‚4805 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.006570411380380392 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8952308893203735 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.622810959815979 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9856441020965576 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.11502408981323242 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.09878669679164886 particle/accuracy_1_of_1‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7799999713897705 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02359246462583542 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.11756092309951782 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3788035809993744 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03919782117009163 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8969069719314575 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988510608673096 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.11829479783773422 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6933782696723938 /particle/accuracy_3_of_4‚0.9955229759216309 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 particle/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8563199639320374 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.08342001587152481 +jet/accuracy_2_of_3‚0.47508883476257324 particle/accuracy_3_of_5‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.023766914382576942 particle/accuracy_1_of_5‚1 epoch‚5 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.941826343536377 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1313161551952362 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7799999713897705 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0714285746216774 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.43692535161972046 *jet/accuracy_1_of_1‚0.2743188142776489 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5752818584442139 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.11638996750116348 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7523455619812012 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 _runtime‚ 2465.84725494 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40909090638160706 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6413089036941528 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6405024528503418 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6137741804122925 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.20901896059513092 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6171441674232483 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6080068945884705 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7528355121612549 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6868059635162354 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6507748365402222 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03619508445262909 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6915391087532043 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 .particle/accuracy_3_of_3‚0.999514102935791 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09071813523769379 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6137741804122925 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49000000953674316 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3194444477558136 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6477194428443909 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12234789878129959 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0035892738960683346 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8355262875556946 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.545059323310852 particle/accuracy_2_of_2‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.10435999929904938 +jet/accuracy_3_of_4‚0.09441577643156052 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2511190176010132 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9957055449485779 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13124999403953552 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6839740872383118 particle/accuracy_1_of_2‚1 % _timestamp‚1.7844889892716777e+09 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8166688084602356 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28377479314804077 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6105924844741821 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.663498044013977 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02767857164144516 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3682684898376465 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3561813235282898 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6118217706680298 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.530468761920929 +jet/accuracy_2_of_2‚0.13125984370708466 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992743134498596 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.10144618898630142 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6985437273979187 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012728306464850903 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.09935101866722107 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1029326319694519 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0714285746216774 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.993574857711792 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6765905618667603 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.917954683303833 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.986619770526886 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9979727864265442 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.996444046497345 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6816357970237732 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.039519548416137695 *particle/sensitivity‚0.625562846660614 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9903395771980286 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10686425119638443 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4249792993068695 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8482905626296997 jet/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9970992207527161 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.16967138648033142 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9961899518966675 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7381882667541504 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9885439872741699 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5897520184516907 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.10606767237186432 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8166688084602356 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.10141504555940628 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7validation_average_jet_accuracy‚0.32340285181999207 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.643570065498352 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6072615385055542 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.10769183188676834 *jet/accuracy_1_of_4‚0.9582488536834717 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8486841917037964 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6713894605636597 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6155487895011902 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.11793318390846252 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +particle/specificity‚0.9032500386238098 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3533458113670349 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8574560880661011 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.7535116076469421 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8504273891448975 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07727242261171341 +jet/accuracy_4_of_4‚0.01666666567325592 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.18092086911201477 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03653086721897125 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.07377177476882935 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 jet/accuracy_2_of_5‚1 jet/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.528556764125824 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10657672584056854 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5573469400405884 'particle/f_score‚0.6320435404777527 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5937500596046448 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6859825849533081 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 (particle/accuracy‚0.8436259031295776 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37654319405555725 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_4‚1 jet/accuracy_1_of_5‚1 )loss/val_total_loss‚8.894206720055362 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4626079797744751 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02767857164144516 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07500000298023224 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7619657516479492 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.879057765007019…f£Y™‚(:Î ¦ÈôÒ¥ó’Ì network.sent‚ 6.481153e+06 network.recv‚685820 disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625# disk.nvme1n1.out‚6922.259765625! disk.dm-0.out‚24048.185546875 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚6922.259765625 disk.md0.in‚0 disk.md0.out‚ 67.94921875 disk.dm-0.in‚ 4.3359375$ memory_percent‚95.48141956329346 proc.cpu.threads‚17) disk./.usagePercent‚76.66470133662321 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1n1.in‚0( proc.memory.availableMB‚ 740.32421875 proc.memory.rssMB‚ 2835.6875* proc.memory.percent‚17.307662963867188% disk./.usageGB‚1176.9948272705078 disk.dm-1.out‚3811.236328125 disk.rbd0.in‚0& disk.nvme0c0n1.out‚21082.818359375 disk.nvme0n1.in‚ 4.3359375$ disk.nvme0n1.out‚21082.818359375 cpu‚26.089051155739156±vžš‚(:“ ¦ÈôÒ ‘ÄÌ gpu.0.gpu‚74.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚253.282' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.42733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™À>2›- _wandbruntime‚2488  _runtime‚2488æ˜PÇȜ ' lr-AdamW/pg2‚0.00020731151479418952 trainer/global_step‚4849 $ _timestamp‚1.784489012307941e+09 _runtime‚2488.883380836 _step‚390 ' lr-AdamW/pg1‚0.00020731151479418952,£œnÍÇ _runtime‚2488.883380836 _step‚390 ' lr-AdamW/pg1‚0.00020731151479418952 ' lr-AdamW/pg2‚0.00020731151479418952 trainer/global_step‚4849 $ _timestamp‚1.784489012307941e+09†ká2ž- _wandbruntime‚2488  _runtime‚2488%L%ä'ŸÞO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚376192459780608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4660447796224 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86897549 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49061881 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚299945658096640 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4652409883914 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚367636790031600 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚376192459780608 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚371532011984384 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚371532029502976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚372289216954874 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚308 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 trainer/global_step‚4849 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49051738 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚376192818085376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 $ _timestamp‚1.784489012308565e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37835668 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37835666 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4660447796224 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚299945500502016 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37835668 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49051738 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 173056512 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚376192818085376 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25762800 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12847632 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ñä¤É ãO KADeviceStafßÀQÖ'tsMonitor.on_train_batch_start/active.small_pool.current‚10143 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚376192459780608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4660447796224 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86897549 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49061881 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚299945658096640 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4652409883914 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚367636790031600 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚376192459780608 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚371532011984384 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚371532029502976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚372289216954874 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚308 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 trainer/global_step‚4849 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49051738 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚376192818085376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 $ _timestamp‚1.784489012308565e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37835668 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37835666 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4660447796224 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚299945500502016 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37835668 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49051738 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 173056512 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚376192818085376 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25762800 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12847632 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0‡îÅ7:2¡- _wandbruntime‚2489  _runtime‚2489Wæï&¢éM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚300005338894336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚2489.255714288 _step‚392 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚372363214297908 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37843168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚371606221272064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37843170 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4661377707520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38617772 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49071770 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚376267581460992 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49061613 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚372363571260702 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚371606221272064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86914940 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚376267939772928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38618075 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4805053575168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4661377707520 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 173056512 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚367710216114080 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49071770 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25767883 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4805186738176 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37843168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚371606221272064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37843170 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4661377707520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38617772 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49071770 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚376267581460992 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49061613 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚372363571260702 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚371606221272064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86914940 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚376267939772928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38618075 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4805053575168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4661377707520 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 173056512 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚367710216114080 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49071770 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25767883 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4805186738176 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚379948576056832 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚375241600438784 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12974237 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4706975618048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4706975618048 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚375241617957376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚379948576056832 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 173056512 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4698852948922 _step‚395 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38991087 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚371308094155600 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38210768 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38991377 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 145011712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚379948934361600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚298085250038272 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49547340 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38210768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12974525 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4852156994048 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚302937527619584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4698513025600 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49557483 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚375241617957376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚371308111194960 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚375241600438784 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚375241600438784 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚379948576056832 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚375241600438784 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12974237 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4706975618048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4706975618048 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚375241617957376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚379948576056832 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 173056512 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4698852948922 _step‚395 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38991087 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚371308094155600 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38210768 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38991377 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 145011712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚379948934361600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚298085250038272 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49547340 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38210768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12974525 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4852156994048 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚302937527619584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4698513025600 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49557483 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚375241617957376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚371308111194960 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 147101696 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49567372 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4699441248836 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87785642 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚302997218538496 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38998710 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚376080961486916 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4707905529344 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4853239269376 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚302997365640192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚298144101946368 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87785642 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38218268 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚380024056049152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 % _timestamp‚1.7844890373876956e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚375315809726464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26021933 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚371381520238080 _step‚396 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12977068 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚298144126370816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38218270 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4853116592128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4707905529344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38999001 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12976779 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26021931 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38218270 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚298144126370816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38218270 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4853116592128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4707905529344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38999001 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12976779 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26021931 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38218270 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 147101696 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49567372 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4699441248836 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87785642 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚302997218538496 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38998710 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚376080961486916 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4707905529344 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4853239269376 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚302997365640192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚298144101946368 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87785642 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38218268 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚380024056049152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 % _timestamp‚1.7844890373876956e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚375315809726464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26021933 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚371381520238080 _step‚396 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12977068 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚383704692333056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4744956090608 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚378951188893184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4753844226048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38585868 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38585866 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4900173307904 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50042942 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38585868 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50042942 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39371564 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚2538.637221616 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4753844226048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13100952 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 trainer/global_step‚4949 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13100664 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88628808 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚305929362657792 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122684416 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88638953 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4900295992320 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚383704692333056 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26270902 _step‚399 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4753503439872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26270900 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4745296013930 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50053085 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚378951188893184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚379724354370208 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50053085 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚378951206411776 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 173056512 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚383705050637824 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚379724711332890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚378951206411776 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚383705050637824 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4753503439872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚305929215548928 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚383704692333056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4744956090608 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚378951188893184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4753844226048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38585868 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38585866 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4900173307904 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50042942 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38585868 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50042942 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39371564 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚2538.637221616 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4753844226048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13100952 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 trainer/global_step‚4949 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13100664 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88628808 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚305929362657792 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122684416 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88638953 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4900295992320 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚383704692333056 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26270902 _step‚399 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4753503439872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26270900 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4745296013930 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50053085 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚378951188893184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚379724354370208 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50053085 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚378951206411776 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 173056512 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 YEDeviceStatsMonitor.on_train_batch_start/allocateT-¸¨Êd_bytes.all.allocated‚383705050637824 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚379724711332890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚378951206411776 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚383705050637824 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4753503439872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚305929215548928 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39379421 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88656344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚301087918573568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4745884313844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 120580096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4754433351168 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39379125 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38593368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚294 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚379025398180864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13103438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88646185 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚379798708675924 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88656344 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 trainer/global_step‚4949 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚379025380662272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50052817 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚400 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚379025398180864 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4754433351168 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚383780172325376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚301087942998016 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4901137141248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4754774144512 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚383780172325376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88646185 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚379799065638718 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚375052824362080 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50052817 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50062974 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚379025380662272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38593370 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4754774144512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 _runtime‚ 2539.00949478 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4901257721344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50062974 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38593368fnÖ¡ó&ÃíM FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4754774144512 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚383780172325376 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88646185 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚379799065638718 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚375052824362080 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50052817 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50062974 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚379025380662272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38593370 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4754774144512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 _runtime‚ 2539.00949478 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4901257721344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50062974 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38593368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚305989055714816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26275983 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26275981 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13103144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7844890624343774e+09 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39379421 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88656344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚301087918573568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4745884313844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 120580096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4754433351168 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39379125 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38593368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚294 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚379025398180864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13103438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88646185 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚379798708675924 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88656344 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 trainer/global_step‚4949 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚379025380662272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50052817 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚400 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚379025398180864 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4754433351168 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚383780172325376 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚301087942998016 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4901137141248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0WÅ «2Ä- _wandbruntime‚2539  _runtime‚2539Ó³I‘ Å„ /loss/FRt2/assignment_loss‚2.115164551635583 2loss/SRqqt1/assignment_loss‚0.6286564320480065 2loss/SRqqt2/assignment_loss‚0.6795681470492542 /loss/FBt2/detection_loss‚0.6845933935809426 trainer/global_step‚4949 3loss/FBt1/assignment_loss‚0.0026147654912429415 4loss/FBt2/assignment_loss‚0.00039735120495230957 1loss/SRqqt1/detection_loss‚0.7432782518781382 /loss/FBt1/detection_loss‚0.6752552247569084 % _timestamp‚1.7844890624354272e+09 0loss/FRt1/assignment_loss‚1.1411273238413475 /loss/FRt1/detection_loss‚1.4113454947809634 1loss/SRqqt2/detection_loss‚0.9696800188230188 epoch‚6 /loss/FRt2/detection_loss‚0.4980230280509507 $loss/total_loss‚9.54970398314131 _runtime‚2539.010078791 _step‚401>ë;MƉ _step‚401 /loss/FRt2/assignment_loss‚2.115164551635583 2loss/SRqqt1/assignment_loss‚0.6286564320480065 2loss/SRqqt2/assignment_loss‚0.6795681470492542 /loss/FBt2/detection_loss‚0.6845933935809426 trainer/global_step‚4949 3loss/FBt1/assignment_loss‚0.0026147654912429415 4loss/FBt2/assignment_loss‚0.00039735120495230957 1loss/SRqqt1/detection_loss‚0.7432782518781382 /loss/FBt1/detection_loss‚0.6752552247569084 % _timestamp‚1.7844890624354272e+09 0loss/FRt1/assignment_loss‚1.1411273238413475 /loss/FRt1/detection_loss‚1.4113454947809634 1loss/SRqqt2/detection_loss‚0.9696800188230188 epoch‚6 /loss/FRt2/detection_loss‚0.4980230280509507 $loss/total_loss‚9.54970398314131 _runtime‚2539.010078791‘ŒQQÇ‚(:Æ ñÈôÒßÎŒÌ network.sent‚ 6.672292e+06% disk./.usageGB‚1173.9912796020508# disk.dm-0.out‚24750.22705078125 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625$ disk.nvme0c0n1.out‚ 21753.3984375$ memory_percent‚96.98657989501953) proc.memory.percent‚17.30804443359375 disk.dm-1.out‚ 4057.828125" disk.nvme0n1.out‚ 21753.3984375 proc.memory.rssMB‚2835.75 cpu‚26.08205918604352% disk.nvme1n1.out‚7204.96044921875 disk.md0.in‚0 disk.md0.out‚ 70.09765625 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375% proc.memory.availableMB‚ 493.71875 proc.cpu.threads‚17 network.recv‚705157) disk./.usagePercent‚76.46906234177176 disk.nvme1n1.in‚0 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚7204.96044921875f1šwžÈ‚(:“ ñÈôÒ©€ÈÌ gpu.0.gpu‚80.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚52.0 gpu.0.powerWatts‚129.578' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚43.19266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0• Ô’2É- _wandbruntime‚2563  _runtime‚2563‚~—±ÉÊà _step‚402 ' lr-AdamW/pg1‚0.00020689744393108132 ' lr-AdamW/pg2‚0.00020689744393108132 trainer/global_step‚4999 % _timestamp‚1.7844890869918737e+09 _runtime‚2563.566850881ZBV›ÎËÈ ' lr-AdamW/pg1‚0.00020689744393108132 ' lr-AdamW/pg2‚0.00020689744393108132 trainer/global_step‚4999 % _timestamp‚1.7844890869918737e+09 _runtime‚2563.566850881 _step‚402’ ’M2Ì- _wandbruntime‚2563  _runtime‚2563oãEŽŒÍßO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26524952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚382660777347584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13227615 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚308921148577280 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚387460808609280 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚382660777347584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚403 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50538544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚4999 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚303972858868224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4948136308736 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38960968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚378650719442960 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89509655 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚303972883292672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4¢¦¨(Y 791739078938 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89499510 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38960968 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38960966 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 173056512 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4800031261696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50548687 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4948265284608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50548687 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4800372047872 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4800372047872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0µeVê'ÎäO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38960968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚378650719442960 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89509655 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚303972883292672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4791739078938 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89499510 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38960968 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38960966 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 173056512 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4800031261696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50548687 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4948265284608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50548687 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4800372047872 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4800372047872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26524952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚382660777347584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13227615 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚308921148577280 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚387460808609280 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚382660777347584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚403 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50538544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚4999 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚303972858868224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4948136308736 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145“}åæ2Ï- _wandbruntime‚2563  _runtime‚2563òi$ìï&ÐéM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚304031759625216 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4949226415616 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135260160 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4792327378852 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4792667302286 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89527046 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 % _timestamp‚1.7844890873712044e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159684608 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚383516812827726 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39759885 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚387535930289664 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50558576 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13230155 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚308980826356224 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50548419 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39760188 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚382734969116672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89527046 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4800961172992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38968470 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚382734986635264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4800961172992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚382734986635264 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38968468 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26530031 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13229854 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚304031735200768 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4949091155456 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚383516455864932 _runtime‚2563.946375416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚387536288601600 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26530033 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50558576 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89516887 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4801301966336 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50548419 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 trainer/global_step‚4999 _step‚404 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚387536288601600 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚382734969116672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794›"[0y$ÑîM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚387535930289664 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50558576 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13230155 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚308980826356224 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50548419 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39760188 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚382734969116672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89527046 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4800961172992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38968470 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚382734986635264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4800961172992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚382734986635264 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38968468 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26530031 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13229854 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚304031735200768 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4949091155456 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚383516455864932 _runtime‚2563.946375416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚387536288601600 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26530033 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50558576 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89516887 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4801301966336 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50548419 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 trainer/global_step‚4999 _step‚404 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚387536288601600 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚382734969116672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚304031759625216 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4949226415616 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135260160 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4792327378852 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4792667302286 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89527046 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 % _timestamp‚1.7844890873712044e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159684608 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚383516812827726 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_‚­¡ô{end/active_bytes.small_pool.current‚ 340793344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39759885 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675”‚Íá2Ò- _wandbruntime‚2563  _runtime‚2563ù9ÞýÓ‚ /loss/FBt2/detection_loss‚0.6334437406376788 epoch‚6 0loss/FRt1/assignment_loss‚1.1043938637252837 2loss/SRqqt1/assignment_loss‚0.6288538599118083 2loss/SRqqt2/assignment_loss‚0.6818129012883274 /loss/FRt2/detection_loss‚0.5682641430636401 1loss/SRqqt1/detection_loss‚0.6787322763358954 /loss/FBt1/detection_loss‚0.7620793145948223 %loss/total_loss‚9.566946686677747 trainer/global_step‚4999 1loss/SRqqt2/detection_loss‚0.9064589413616694 0loss/FRt2/assignment_loss‚1.9653936259682403 2loss/FBt1/assignment_loss‚0.036803596876687696 1loss/FBt2/assignment_loss‚0.11648052254211494 /loss/FRt1/detection_loss‚1.4842299003715789 % _timestamp‚1.7844890873723295e+09 _runtime‚2563.947029937 _step‚405² Ô‡ /loss/FBt2/detection_loss‚0.6334437406376788 epoch‚6 0loss/FRt1/assignment_loss‚1.1043938637252837 2loss/SRqqt1/assignment_loss‚0.6288538599118083 2loss/SRqqt2/assignment_loss‚0.6818129012883274 /loss/FRt2/detection_loss‚0.5682641430636401 1loss/SRqqt1/detection_loss‚0.6787322763358954 /loss/FBt1/detection_loss‚0.7620793145948223 %loss/total_loss‚9.566946686677747 trainer/global_step‚4999 1loss/SRqqt2/detection_loss‚0.9064589413616694 0loss/FRt2/assignment_loss‚1.9653936259682403 2loss/FBt1/assignment_loss‚0.036803596876687696 1loss/FBt2/assignment_loss‚0.11648052254211494 /loss/FRt1/detection_loss‚1.4842299003715789 % _timestamp‚1.7844890873723295e+09 _runtime‚2563.947029937 _step‚405•%¢°²[Õ‚(:Ð €ÉôÒ‹´õË) proc.memory.percent‚17.30804443359375 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0 network.recv‚709678 disk.rbd0.in‚0 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚25120.25927734375 disk.dm-1.in‚0' disk.nvme0c0n1.out‚22123.9365234375% disk.nvme0n1.out‚22123.9365234375 disk.md0.out‚ 70.6796875 disk.dm-1.out‚ 4058.08984375$ memory_percent‚96.98693752288818( proc.memory.availableMB‚ 493.66015625 cpu‚26.082845935684283' disk.nvme1c1n1.out‚7205.94677734375 disk.nvme1n1.in‚0% disk.nvme1n1.out‚7205.94677734375 network.sent‚ 6.734642e+06) disk./.usagePercent‚76.46927205383768$ disk./.usageGB‚1173.994499206543 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 proc.memory.rssMB‚2835.75ærEnÖ‚(:’ €ÉôÒ›ê¨Ì gpu.0.gpu‚90.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚52.0 gpu.0.powerWatts‚119.0' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.666666666666664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.§ÊÞ^ׂ(:Ó ÉôÒÙÓË network.recv‚713876% disk./.usageGB‚1173.9942741394043 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚25121.25927734375" disk.dm-1.out‚4058.27001953125 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚7206.63330078125 disk.nvme1n1.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚22124.93505859375 disk.nvme0n1.in‚ 4.3359375$ memory_percent‚96.98741436004639 proc.memory.rssMB‚2835.75 disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375% disk.nvme1n1.out‚7206.63330078125 cpu‚26.08468565945122 network.sent‚ 6.784333e+06) disk./.usagePercent‚76.46925739387098 disk.md0.in‚0 disk.md0.out‚70.90625& disk.nvme0n1.out‚22124.93505859375( proc.memory.availableMB‚ 493.58203125) proc.memory.percent‚17.30804443359375 proc.cpu.threads‚173´±ŸØ‚(:” ÉôÒõðÌ gpu.0.gpu‚80.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚51.0 gpu.0.powerWatts‚117.655' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.218333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&´‡â2Ù- _wandbruntime‚2588  _runtime‚2588úÚÆÚÀ trainer/global_step‚5049 $ _timestamp‚1.784489111750627e+09 _runtime‚2588.325574401 _step‚406 & lr-AdamW/pg1‚0.0002067561987844627 & lr-AdamW/pg2‚0.0002067561987844627?·ËÛÅ & lr-AdamW/pg1‚0.0002067561987844627 & lr-AdamW/pg2‚0.0002067561987844627 trainer/global_step‚5049 $ _timestamp‚1.784489111750627e+09 _runtime‚2588.325574401 _step‚406–º-,=2Ü- _wandbruntime‚2588  _runtime‚2588r‰Áå'ÝßO G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 trainer/global_step‚5049 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4838182143946 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39336066 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90370212 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚382322023566960 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚386370365801984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚386370365801984 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚90380357 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚387160205710906 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4846559083520 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚382322006527600 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4846559083520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚305 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 % _timestamp‚1.7844891117515614e+09 _step‚407 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4846899869696 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39336066 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4996207453184 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚391216924885504 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26779002 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4996074283008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚386370383320576 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39336068 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚391216924885504 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26779002 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4996074283008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚386370383320576 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39336068 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚90380357 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚387160205710906 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4846559083520 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚382322006527600 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4846559083520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚305 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 % _timestamp‚1.7844891117515614e+09 _step‚407 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4846899869696 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39336066 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4996207453184 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675—ÎZµw2ß- _wandbruntime‚2588  _runtime‚2588©tãàéM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 _step‚408 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4839110367294 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚391292404877824 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90387589 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚387234203053940 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 173056512 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 _runtime‚2588.687131276 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚391292404877824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚386444575089664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13356867 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚387234560016734 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚311972580048384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39343570 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13356565 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4997169772032 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90397748 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51044021 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4847829788160 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚306975551827968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40140950 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26784081 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚382395432610080 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51054178 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39343568 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4847829788160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4847488994816 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚386444557571072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26784083 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚5049 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141551616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 % _timestamp‚1.7844891121114194e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4847488994816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚302 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39343570 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4838770443860 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 165976064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51044021 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51054178 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚311972746024448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40140646#6ô&áîM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51044021 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51054178 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚311972746024448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40140646 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 _step‚408 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4839110367294 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚391292404877824 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90387589 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚387234203053940 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 173056512 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 _runtime‚2588.687131276 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚391292404877824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚386444575089664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13356867 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚387234560016734 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚311972580048384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39343570 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13356565 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4997169772032 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90397748 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51044021 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4847829788160 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚306975551827968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40140950 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26784081 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚382395432610080 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51054178 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39343568 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4847829788160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4847488994816 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚386444557571072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26784083 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚5049 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141551616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 % _timestamp‚1.7844891121114194e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4847488994816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚302 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39343570 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4838770443860 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 165976064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260˜ŠÌH{2â- _wandbruntime‚2588  _runtime‚2588¸•)ã‚ $ _timestamp‚1.784489112112372e+09 _step‚409 /loss/FRt2/detection_loss‚0.5367700749385726 1loss/SRqqt1/detection_loss‚0.6155945307502758 1loss/SRqqt2/detection_loss‚1.2264304119664773 /loss/FBt2/detection_loss‚0.6157388831178348 epoch‚6 _runtime‚2588.687957367 2loss/SRqqt2/assignment_loss‚0.8621828485106071 2loss/FBt1/assignment_loss‚0.011338153689835342 /loss/FBt1/detection_loss‚0.6733956469685428 0loss/FRt2/assignment_loss‚2.1880626069174873 2loss/SRqqt1/assignment_loss‚0.5412660109134121 0loss/FRt1/assignment_loss‚1.0690914394654634 2loss/FBt2/assignment_loss‚0.021173419361464833 /loss/FRt1/detection_loss‚1.3649074880955503 %loss/total_loss‚9.725951514695524 trainer/global_step‚5049Ÿ›m( ä‡ 0loss/FRt1/assignment_loss‚1.0690914394654634 2loss/FBt2/assignment_loss‚0.021173419361464833 /loss/FRt1/detection_loss‚1.3649074880955503 %loss/total_loss‚9.725951514695524 trainer/global_step‚5049 $ _timestamp‚1.784489112112372e+09 _step‚409 /loss/FRt2/detection_loss‚0.5367700749385726 1loss/SRqqt1/detection_loss‚0.6155945307502758 1loss/SRqqt2/detection_loss‚1.2264304119664773 /loss/FBt2/detection_loss‚0.6157388831178348 epoch‚6 _runtime‚2588.687957367 2loss/SRqqt2/assignment_loss‚0.8621828485106071 2loss/FBt1/assignment_loss‚0.011338153689835342 /loss/FBt1/detection_loss‚0.6733956469685428 0loss/FRt2/assignment_loss‚2.1880626069174873 2loss/SRqqt1/assignment_loss‚0.5412660109134121™Ùæ¸üHå‚(:½ žÉôÒ…œòË proc.cpu.threads‚17$ disk./.usageGB‚1171.308006286621 disk.md0.in‚0" disk.dm-1.out‚4058.40673828125 disk.rbd0.in‚0% proc.memory.availableMB‚ 493.53125 network.recv‚717531 disk.dm-0.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 disk.nvme1n1.out‚ 7215.8125 proc.memory.rssMB‚2835.75) proc.memory.percent‚17.30804443359375) disk./.usagePercent‚76.29428472800008# disk.dm-0.out‚25141.68408203125 disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625% disk.nvme0c0n1.out‚22137.46484375# disk.nvme0n1.out‚22137.46484375 disk.nvme1c1n1.out‚ 7215.8125$ memory_percent‚96.98772430419922 network.sent‚ 6.790355e+06 disk.md0.out‚ 71.4453125 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 cpu‚26.08773936297637©’þ6œæ‚(:‘ žÉôÒãƒÓÌ gpu.0.gpu‚4.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚251.933' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚83.97766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü… Qç‚(:Æ ­ÉôÒ©ÐóË network.recv‚721717) disk./.usagePercent‚76.41599574707905% disk./.usageGB‚1173.1765747070312# disk.dm-0.out‚25332.43896484375 disk.nvme0n1.in‚ 4.3359375 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 7216.55078125 proc.memory.rssMB‚2835.75 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme1c1n1.in‚0& proc.memory.availableMB‚ 493.515625 proc.cpu.threads‚17 network.sent‚ 6.85344e+06 disk.md0.out‚71.90625 disk.nvme0c0n1.in‚ 4.3359375$ disk.nvme1c1n1.out‚ 7216.55078125) proc.memory.percent‚17.30804443359375 cpu‚26.09162014184309" disk.dm-1.out‚4058.55419921875 disk.rbd0.in‚0& disk.nvme0c0n1.out‚22328.615234375$ disk.nvme0n1.out‚22328.615234375$ memory_percent‚96.98781967163086%±žè‚(:“ ­ÉôÒ—œ¤Ì gpu.0.gpu‚56.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚263.386' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.79533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x½"w2é- _wandbruntime‚2612  _runtime‚2612áï1½ÇêÁ & lr-AdamW/pg2‚0.0002066133466241132 trainer/global_step‚5099 % _timestamp‚1.7844891361218855e+09 _runtime‚2612.696853872 _step‚410 & lr-AdamW/pg1‚0.0002066133466241132í(sŸÌëÆ & lr-AdamW/pg2‚0.0002066133466241132 trainer/global_step‚5099 % _timestamp‚1.7844891361218855e+09 _runtime‚2612.696853872 _step‚410 & lr-AdamW/pg1‚0.0002066133466241132šä$‰¨2ì- _wandbruntime‚2612  _runtime‚2612r©œ1å'íßO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚309860492122624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39711166 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39711168 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚390877595937232 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4884285285632 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91251059 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5044261346304 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122684416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 173056512 % _timestamp‚1.7844891361228807e+09 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚295 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4893427691520 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚297 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚394973041161728 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚390079954256384 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚394973041161728 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚390079971774976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 _runtime‚2612.697805623 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚314904630784512 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4893086905344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚314904777893376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚390079954256384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚385993327690960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4884625208954 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40513789 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚390079954256384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚385993327690960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4884625208954 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40513789 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39711166 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39711168 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚390877595937232 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4884285285632 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91251059 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5044261346304 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122684416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 173056512 % _timestamp‚1.7844891361228807e+09 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚295 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4893427691520 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚297 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚394973041161728 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚390079954256384 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚394973041161728 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚390079971774976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 _runtime‚2612.697805623 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚314904630784512›@Xpð2ï- _wandbruntime‚2613  _runtime‚2613¥Pàï&ðéM YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40521411 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5045219282432 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40521704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39718670 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91258291 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 trainer/global_step‚5099 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚390154163544064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 _runtime‚2613.068781673 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4894016816640 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚309919392879616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39718668 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91268450 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4885553432302 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51539623 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91258291 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5045092410880 _step‚412 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4894357609984 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4894357609984 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51549780 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚386066753773440 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚386066736734080 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚390154163544064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚390951950242948 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚309919368455168 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚390154146025472 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚395048521154048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13483280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 173056512j|oô&ñîM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51539623 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91258291 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5045092410880 _step‚412 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4894357609984 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4894357609984 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51549780 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚386066753773440 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚386066736734080 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚390154163544064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚390951950242948 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚309919368455168 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚390154146025472 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚395048521154048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13483280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 173056512 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40521411 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5045219282432 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40521704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39718670 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91258291 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 trainer/global_step‚5099 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚390154163544064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 _runtime‚2613.068781673 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4894016816640 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚309919392879616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39718668 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91268450 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4885553432302 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240œ¿pƒ2ò- _wandbruntime‚2613  _runtime‚2613«Q£Wóü 0loss/SRqqt2/detection_loss‚0.880125750677268 /loss/FRt2/assignment_loss‚2.036630759809328 1loss/FBt1/assignment_loss‚0.02243985133806334 /loss/FBt1/detection_loss‚0.6353407977462316 $loss/total_loss‚8.92055278690679 _runtime‚2613.069729895 2loss/SRqqt2/assignment_loss‚0.6787466726954766 .loss/FRt2/detection_loss‚0.565872933441563 /loss/FBt2/detection_loss‚0.6255506052076817 $ _timestamp‚1.784489136494685e+09 2loss/FBt2/assignment_loss‚0.000350969108553727 1loss/SRqqt1/detection_loss‚0.6873126339847115 epoch‚6 trainer/global_step‚5099 _step‚413 0loss/FRt1/assignment_loss‚0.9806432145548043 2loss/SRqqt1/assignment_loss‚0.5150720099800833 .loss/FRt1/detection_loss‚1.292466588363024P²†ô 2loss/SRqqt2/assignment_loss‚0.6787466726954766 .loss/FRt2/detection_loss‚0.565872933441563 /loss/FBt2/detection_loss‚0.6255506052076817 $ _timestamp‚1.784489136494685e+09 2loss/FBt2/assignment_loss‚0.000350969108553727 1loss/SRqqt1/detection_loss‚0.6873126339847115 epoch‚6 trainer/global_step‚5099 _step‚413 0loss/FRt1/assignment_loss‚0.9806432145548043 2loss/SRqqt1/assignment_loss‚0.5150720099800833 .loss/FRt1/detection_loss‚1.292466588363024 0loss/SRqqt2/detection_loss‚0.880125750677268 /loss/FRt2/assignment_loss‚2.036630759809328 1loss/FBt1/assignment_loss‚0.02243985133806334 /loss/FBt1/detection_loss‚0.6353407977462316 $loss/total_loss‚8.92055278690679 _runtime‚2613.069729895}q¿Zõ‚(:Ï ¼ÉôÒ†›ÉË proc.memory.rssMB‚2835.75 proc.cpu.threads‚17 network.sent‚ 6.856362e+06 network.recv‚725281) disk./.usagePercent‚76.47416028307113& disk.nvme0c0n1.out‚24060.443359375& disk.nvme1c1n1.out‚7218.5849609375 disk.md0.in‚0# disk.dm-0.out‚27065.17431640625 disk.dm-1.in‚0% disk./.usageGB‚1174.0695457458496 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375$ disk.nvme0n1.out‚24060.443359375 disk.nvme1c1n1.in‚0 disk.md0.out‚ 72.3828125" disk.dm-1.out‚4058.67529296875 disk.rbd0.out‚ 0.4697265625 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚7218.5849609375 disk.rbd0.in‚0' proc.memory.availableMB‚ 493.8359375) proc.memory.percent‚17.30804443359375 cpu‚26.095179920083105 disk.dm-0.in‚ 4.3359375$ memory_percent‚96.98586463928223tV–ãö‚(:’ ¼ÉôÒ¥”¥Ì gpu.0.gpu‚12.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚241.996' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚80.66533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,Èzº2÷- _wandbruntime‚2638  _runtime‚2638f;PÈø ' lr-AdamW/pg2‚0.00020646888973810008 trainer/global_step‚5149 % _timestamp‚1.7844891617830377e+09 _runtime‚ 2638.35802883 _step‚414 ' lr-AdamW/pg1‚0.00020646888973810008²\O÷ÍùÇ trainer/global_step‚5149 % _timestamp‚1.7844891617830377e+09 _runtime‚ 2638.35802883 _step‚414 ' lr-AdamW/pg1‚0.00020646888973810008 ' lr-AdamW/pg2‚0.00020646888973810008ž®<6Ä2ú- _wandbruntime‚2638  _runtime‚2638?Ç×ÖûßO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40086266 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚310 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27287102 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚398729157437952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13608180 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚393789560229376 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚394595343126240 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52025350 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5092284669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 173056512 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚389664614775600 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4931068273962 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚393789560229376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92111616 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚393789542710784 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4939955513344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚312804333174272 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚394595700088922 trainer/global_step‚5149 % _timestamp‚1.7844891617840836e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147108864 _runtime‚2638.362462597 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚389664631814960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 5452595@ýÎoÐ20 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40086268 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40895280 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚317896617844224 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚393789542710784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27287100 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚308 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122684416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40086266 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚398729515742720 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚398729515742720 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4939955513344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5092161985536 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4939614727168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92121761 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚398729157437952 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚312804308749824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚398729157437952 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚312804308749824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚310 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27287102 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚398729157437952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13608180 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚393789560229376 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚394595343126240 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52025350 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5092284669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 173056512 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚389664614775600 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4931068273962 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚393789560229376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92111616 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚393789542710784 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4939955513344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚312804333174272 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚394595700088922 trainer/global_step‚5149 % _timestamp‚1.7844891617840836e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147108864 _runtime‚2638.362462597 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚389664631814960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40086268 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40895280 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚317896617844224 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚393789542710784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27287100 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚308 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122684416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40086266 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚398729515742720 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚398729515742720 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4939955513344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5092161985536 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4939614727168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92121761 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40093768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚394670054394750 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40093768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13610988 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52045382 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13610693 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚389738057897440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4931996497310 trainer/global_step‚5149 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚398804637430272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚398804279118336 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40093770 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚317956453042688 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 173056512 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4940885431808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚297 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 _runtime‚2638.736453604 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92128993 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5093116664320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚393863734479872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚393863751998464 _step‚416 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40902874 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4931656573876 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40093770 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52035225 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚398804637430272 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27292181 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚317956301746688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5093243535872 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92128993 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92139152 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52035225 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260&÷& ÿíM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚389738057897440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4931996497310 trainer/global_step‚5149 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚398804637430272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚398804279118336 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40093770 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚317956453042688 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 173056512 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4940885431808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚297 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 _runtime‚2638.736453604 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92128993 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5093116664320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚393863734479872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚393863751998464 _step‚416 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40902874 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4931656573876 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40093770 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52035225 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚398804637430272 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27292181 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚317956301746688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5093243535872 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92128993 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92139152 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52035225 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40093768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚394670054394750 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40093768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13610988 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52045382 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13610693 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 úü|à2€- _wandbruntime‚2638  _runtime‚2638'\ÿ 0loss/FRt2/assignment_loss‚2.1126695421530153 /loss/FRt2/detection_loss‚0.5451861036860904 1loss/SRqqt1/detection_loss‚0.7228130681993629 %loss/total_loss‚9.511706718358884 epoch‚6 trainer/global_step‚5149 2loss/SRqqt1/assignment_loss‚0.6235212307041207 2loss/SRqqt2/assignment_loss‚0.6661048502386581 /loss/FRt1/detection_loss‚1.3928941796049201 % _timestamp‚1.7844891621623926e+09 _step‚417 1loss/FBt1/assignment_loss‚0.05771608042480005 1loss/FBt2/assignment_loss‚0.26485440336028104 /loss/SRqqt2/detection_loss‚1.06503972216851 /loss/FBt1/detection_loss‚0.5208217451296184 /loss/FBt2/detection_loss‚0.6303855991611879 _runtime‚2638.740391391 0loss/FRt1/assignment_loss‚0.9097001935283197å'Ê ‚„ % _timestamp‚1.7844891621623926e+09 _step‚417 1loss/FBt1/assignment_loss‚0.05771608042480005 1loss/FBt2/assignment_loss‚0.26485440336028104 /loss/SRqqt2/detection_loss‚1.06503972216851 /loss/FBt1/detection_loss‚0.5208217451296184 /loss/FBt2/detection_loss‚0.6303855991611879 _runtime‚2638.740391391 0loss/FRt1/assignment_loss‚0.9097001935283197 0loss/FRt2/assignment_loss‚2.1126695421530153 /loss/FRt2/detection_loss‚0.5451861036860904 1loss/SRqqt1/detection_loss‚0.7228130681993629 %loss/total_loss‚9.511706718358884 epoch‚6 trainer/global_step‚5149 2loss/SRqqt1/assignment_loss‚0.6235212307041207 2loss/SRqqt2/assignment_loss‚0.6661048502386581 /loss/FRt1/detection_loss‚1.3928941796049201¡ÞÕ>ñTƒ‚(:É ËÉôÒÕ¤áË) proc.memory.percent‚17.30804443359375 cpu‚26.09897130882997# disk.nvme0c0n1.out‚ 24061.171875 disk.nvme0n1.in‚ 4.3359375 disk.md0.in‚0# disk.dm-0.out‚27065.70166015625 disk.rbd0.out‚ 0.4697265625 network.sent‚ 6.931623e+06 network.recv‚728599$ disk./.usageGB‚1174.069751739502 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚7218.99169921875 disk.rbd0.in‚0 proc.cpu.threads‚17! disk.nvme0n1.out‚ 24061.171875 disk.nvme1n1.in‚0% disk.nvme1n1.out‚7218.99169921875 disk.md0.out‚ 72.63671875 disk.dm-1.in‚0 disk.dm-1.out‚4058.740234375( proc.memory.availableMB‚ 493.77734375 proc.memory.rssMB‚2835.75) disk./.usagePercent‚76.47417370066776 disk.dm-0.in‚ 4.3359375$ memory_percent‚96.98622226715088•d–Vž„‚(:“ ËÉôÒª¾—Ì gpu.0.gpu‚65.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚252.061' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.02033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NHŒÙZ…‚(:Ï ÚÉôÒ겂Ì% disk./.usageGB‚1176.0240478515625 disk.nvme0c0n1.in‚ 4.3359375% disk.nvme1n1.out‚7295.80419921875 disk.rbd0.out‚ 0.4697265625$ memory_percent‚96.98481559753418) proc.memory.percent‚17.30804443359375 disk.nvme0n1.in‚ 4.3359375 disk.md0.out‚ 72.90234375 proc.memory.rssMB‚2835.75 cpu‚26.09887811593929 proc.cpu.threads‚17$ disk.nvme0n1.out‚25099.857421875 disk.nvme1n1.in‚0 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚28180.48095703125 disk.dm-1.in‚0' proc.memory.availableMB‚ 494.0078125 network.sent‚ 6.981882e+06) disk./.usagePercent‚76.60146867621316& disk.nvme0c0n1.out‚25099.857421875 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚7295.80419921875 disk.md0.in‚0 disk.dm-1.out‚4058.880859375 disk.rbd0.in‚0 network.recv‚7331385u‘Áž†‚(:“ ÚÉôÒ…Æ·Ì gpu.0.gpu‚86.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚243.044' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚81.01466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘ â¯2‡- _wandbruntime‚2663  _runtime‚2663};*Ɉà ' lr-AdamW/pg1‚0.00020632283044019363 ' lr-AdamW/pg2‚0.00020632283044019363 trainer/global_step‚5199 % _timestamp‚1.7844891866539414e+09 _runtime‚2663.228859902 _step‚418~m:Î‰È ' lr-AdamW/pg2‚0.00020632283044019363 trainer/global_step‚5199 % _timestamp‚1.7844891866539414e+09 _runtime‚2663.228859902 _step‚418 ' lr-AdamW/pg1‚0.00020632283044019363¢ø®Ñ2Š- _wandbruntime‚2663  _runtime‚2663f2ƒ˜å'‹ßO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92992463 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41275862 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 173056512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚398313447277930 trainer/global_step‚5199 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27541150 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4986483335168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122684416 _runtime‚2663.232456389 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 _step‚419 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41276156 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚393335935938960 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40461368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13735004 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5140291953664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚397499131165184 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13734712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚402485273714176 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚402485632018944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚315748149801472 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52531095 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40461366 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚315748125377024 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4977511338970 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4986483335168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27541152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚393335918899600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52520952 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚402485273714176 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4986142548992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52520952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92982318 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚397499148683776 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚320888294646272 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚402485632018944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4986142548992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚402485273714176 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4986142548992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52520952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92982318 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚397499148683776 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚320888294646272 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚402485632018944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4986142548992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚402485273714176 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚402485632018944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚315748149801472 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52531095 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40461366 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚315748125377024 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4977511338970 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4986483335168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27541152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚393335918899600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52520952 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 147101696 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚315807001709568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 173056512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4987413253632 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4987072460288 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚397573340452864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4987072460288 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92999695 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5141255398912 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚402560395394560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4987413253632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4978099638884 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93009854 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40468870 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52530827 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 \JDeviceStatsMonitor.on_train¦ ÀÙv_batch_end/requested_bytes.small_pool.allocated‚ 4978439562318 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13737158 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 _step‚420 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5141132721664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚398387801583758 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 trainer/global_step‚5199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52530827 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚397573322934272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚393409362021440 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚397573322934272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40468868 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122677248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92999695 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52540984 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40468868 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚397573340452864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41283389 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚393409344982080 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40468870 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚402560753706496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41283681 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 % _timestamp‚1.7844891870116644e+09 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚402560395394560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4987413253632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4978099638884 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93009854 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40468870 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52530827 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4978439562318 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13737158 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 _step‚420 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5141132721664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚398387801583758 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 trainer/global_step‚5199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52530827 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚397573322934272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚393409362021440 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚397573322934272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40468868 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122677248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92999695 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52540984 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40468868 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚397573340452864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41283389 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚393409344982080 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40468870 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚402560753706496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41283681 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 % _timestamp‚1.7844891870116644e+09 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 147101696 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚315807001709568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 173056512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4987413253632 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4987072460288 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚397573340452864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4987072460288 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2¤7-2- _wandbruntime‚2663  _runtime‚2663^x(‘ý /loss/FRt2/assignment_loss‚2.101006169813498 1loss/SRqqt1/assignment_loss‚0.592598108706149 trainer/global_step‚5199 /loss/FRt1/assignment_loss‚0.941644394670734 2loss/FBt1/assignment_loss‚0.010763051979037784 2loss/FBt2/assignment_loss‚0.030340762056291187 /loss/FRt1/detection_loss‚1.4573931624217116 1loss/SRqqt1/detection_loss‚0.7090519434711858 1loss/SRqqt2/detection_loss‚1.1416344044005706 /loss/FBt1/detection_loss‚0.8171208544773981 _runtime‚2663.587306065 1loss/SRqqt2/assignment_loss‚0.802731886776854 /loss/FBt2/detection_loss‚0.6505685810766676 _step‚421 %loss/total_loss‚9.811068903277672 epoch‚6 /loss/FRt2/detection_loss‚0.5562155834275739 # _timestamp‚1.78448918701259e+09·C´´’‚ /loss/FBt1/detection_loss‚0.8171208544773981 _runtime‚2663.587306065 1loss/SRqqt2/assignment_loss‚0.802731886776854 /loss/FBt2/detection_loss‚0.6505685810766676 _step‚421 %loss/total_loss‚9.811068903277672 epoch‚6 /loss/FRt2/detection_loss‚0.5562155834275739 # _timestamp‚1.78448918701259e+09 /loss/FRt2/assignment_loss‚2.101006169813498 1loss/SRqqt1/assignment_loss‚0.592598108706149 trainer/global_step‚5199 /loss/FRt1/assignment_loss‚0.941644394670734 2loss/FBt1/assignment_loss‚0.010763051979037784 2loss/FBt2/assignment_loss‚0.030340762056291187 /loss/FRt1/detection_loss‚1.4573931624217116 1loss/SRqqt1/detection_loss‚0.7090519434711858 1loss/SRqqt2/detection_loss‚1.1416344044005706¥°µS“‚(:È éÉôÒØ«ïË% disk.nvme0n1.out‚26770.5009765625 disk.nvme1c1n1.in‚0! disk.nvme1c1n1.out‚ 7296.90625 disk.md0.in‚0 proc.memory.rssMB‚2835.75 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 disk.nvme1n1.out‚ 7296.90625 disk.md0.out‚ 73.36328125 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚29851.11474609375" disk.dm-1.out‚4059.00439453125( proc.memory.availableMB‚ 493.97265625 disk.nvme1n1.in‚0 network.sent‚ 6.984804e+06 network.recv‚736420) disk./.usagePercent‚76.65109564516098$ disk./.usageGB‚1176.785945892334 disk.rbd0.out‚ 0.4697265625 disk.dm-1.in‚0) proc.memory.percent‚17.30804443359375 disk.rbd0.in‚0' disk.nvme0c0n1.out‚26770.5009765625$ memory_percent‚96.98503017425537 cpu‚26.097490226056582¬@Žž”‚(:“ éÉôÒïûÓÌ gpu.0.gpu‚92.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚267.877' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.29233333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ß:ïb•‚(:× øÉôÒÊÊìË disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚26777.35986328125$ memory_percent‚96.98503017425537 cpu‚26.095374113162237' disk.nvme1c1n1.out‚7298.90576171875 disk.dm-0.in‚ 4.3359375" disk.dm-1.out‚4059.08251953125 proc.memory.rssMB‚2835.75 proc.cpu.threads‚17) disk./.usagePercent‚76.65111875324408 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0# disk.dm-0.out‚29858.90380859375 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625% disk./.usageGB‚1176.7863006591797% disk.nvme1n1.out‚7298.90576171875 disk.md0.in‚0 disk.md0.out‚ 73.82421875 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 4.3359375( proc.memory.availableMB‚ 493.97265625) proc.memory.percent‚17.30804443359375 network.sent‚ 7.048146e+06 network.recv‚740615( disk.nvme0c0n1.out‚26777.35986328125‡®.“–‚(:ˆ øÉôÒ˜„ºÌ gpu.0.gpu‚91.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚182.184' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚60.728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’Þ‘32—- _wandbruntime‚2687  _runtime‚2687hÌ1šÉ˜Ã ' lr-AdamW/pg1‚0.00020617517106982992 ' lr-AdamW/pg2‚0.00020617517106982992 trainer/global_step‚5249 % _timestamp‚1.7844892105957541e+09 _runtime‚2687.170730443 _step‚422$öN“Î™È ' lr-AdamW/pg1‚0.00020617517106982992 ' lr-AdamW/pg2‚0.00020617517106982992 trainer/global_step‚5249 % _timestamp‚1.7844892105957541e+09 _runtime‚2687.170730443 _step‚422¦*ÝM2š- _wandbruntime‚2687  _runtime‚2687b ÝÚä'›ÞO VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41655840 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚401208719619584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13860354 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚402031194466938 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚401208737138176 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40836468 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93863165 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53016554 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40836468 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13860638 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 _runtime‚2687.171676525 _step‚423 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27795200 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚402030837504256 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚406241389990400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27795202 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5188281683968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚318691966428672 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5033011156992 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53016554 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 173056512 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5023954403978 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚5249 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40836466 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚406241748295168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚401208737138176 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚401208719619584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚397007240062960 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5032670370816 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚397007223023600 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93853020 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚406241389990400 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53026697 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚323880248112640 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚318691942004224 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53026697 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40836466 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5188144319488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚406241389990400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27795202 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5188281683968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚318691966428672 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5033011156992 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53016554 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 173056512 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5023954403978 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚5249 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40836466 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚406241748295168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚401208737138176 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚401208719619584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚397007240062960 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5032670370816 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚397007223023600 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93853020 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚406241389990400 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53026697 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚323880248112640 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚318691942004224 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53026697 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40836466 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5188144319488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0§ëë·W2- _wandbruntime‚2687  _runtime‚26879•ýï&žéM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53036586 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5033941075456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚397080666145440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5033941075456 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5033600282112 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5189234475520 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚401282928907264 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93880556 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚406316869982720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚318750842761216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93880556 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27800283 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚406316511670784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚401282911388672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5033600282112 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41663193 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7844892109542806e+09 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93870397 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5024542703892 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 _runtime‚2687.529433993 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41663477 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚402105191809972 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13863194 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚406316869982720 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚401282928907264 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40843968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5024882627326 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40843970 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53036586 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53026429 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5189109701120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚397080666145440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5033941075456 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5033600282112 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5189234475520 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚401282928907264 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93880556 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚406316869982720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚318750842761216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93880556 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27800283 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚406316511670784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚401282911388672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5033600282112 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41663193 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7844892109542806e+09 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93870397 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5024542703892 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 _runtime‚2687.529433993 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41663477 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚402105191809972 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13863194 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚406316869982720 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚401282928907264 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40843968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5024882627326 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40843970 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53036586 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53026429 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5189109701120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5079538978816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41211566 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5079198192640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41211566 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13987230 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚405748941655946 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41211568 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28049252 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94723722 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53522299 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚326871918325248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚404918325592576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42036779 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚400678544186960 _runtime‚2710.831466744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5079198192640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124781568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 173056512 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5070057545664 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53522299 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚321635783055872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚409997506266624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41211568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5070397468986 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚404918308073984 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚409997506266624 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53512156 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚404918325592576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42036779 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚400678544186960 _runtime‚2710.831466744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5079198192640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124781568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 173056512 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5070057545664 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53522299 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚321635783055872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚409997506266624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41211568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5070397468986 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚404918308073984 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚409997506266624 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53512156 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5079538978816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41211566 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5079198192640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41211566«s`t 2«- _wandbruntime‚2711  _runtime‚2711¡g&†î&¬èM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚404992499843072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 _runtime‚2711.186344602 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5080128103936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5080128103936 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 trainer/global_step‚5299 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41219070 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚404992499843072 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 173056512 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53532188 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41219068 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚321694659388416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5071325692334 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚410072986258944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94741099 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42044360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28054331 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚400751970269440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚321694634963968 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚405823295961774 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42044071 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5237245413888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5080468897280 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5070985768900 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41219068 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚404992517361664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94751258 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53522031 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28054333 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13990027 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94741099 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0©]†§† ­íM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53522031 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚326931751409152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5237116445184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53532188 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.784489234611193e+09 _step‚428 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13989740 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53532188 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41219068 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚321694659388416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5071325692334 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚410072986258944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_b@÷þGmatch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94741099 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42044360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28054331 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚400751970269440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚321694634963968 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚405823295961774 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42044071 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5237245413888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5080468897280 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5070985768900 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41219068 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚404992517361664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94751258 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53522031 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28054333 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13990027 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94741099 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚404992499843072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 _runtime‚2711.186344602 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5080128103936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5080128103936 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 trainer/global_step‚5299 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41219070 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚404992499843072 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 173056512 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚409466688844954 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42416250 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41586666 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 $ _timestamp‚1.784489258316494e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41586668 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5284269666304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 _step‚431 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41586668 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚329863724337664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚324579599683072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54007758 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5125726014464 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚408627896528384 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚413753980847616 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95604569 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28303300 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28303302 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42416543 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚413753622542848 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚324579575258624 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5116500610672 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5126066800640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚329863869349376 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 120587264 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚2734.891349374 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚409466688844954 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42416250 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41586666 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 $ _timestamp‚1.784489258316494e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41586668 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5284269666304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 _step‚431 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41586668 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚329863724337664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚324579599683072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54007758 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5125726014464 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚408627896528384 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚413753980847616 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95604569 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28303300 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28303302 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42416543 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚413753622542848 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚324579575258624 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5116500610672 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5126066800640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚329863869349376 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 120587264 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚2734.891349374 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446¯±ð¹§2»- _wandbruntime‚2735  _runtime‚2735À‹åUô¼éM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54027790 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122677248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚èì¿?û$54017633 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚408702088297472 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚404423274393440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5117768757342 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95611801 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14115707 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 147101696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41594170 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95621960 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 % _timestamp‚1.7844892586729205e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95611801 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54017633 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚408702105816064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚329923705104896 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5117428833908 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41594168 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚408702105816064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5126655925760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚413828744223232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5126655925760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28308381 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚324638451591168 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5285229089280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 _runtime‚2735.248139502 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 173056512 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95621960 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42423799 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚413829102535168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42424090 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14115418 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 trainer/global_step‚5349 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚404423257354080É„€]ô&½îM RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚408702105816064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚329923705104896 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5117428833908 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41594168 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚408702105816064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5126655925760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚413828744223232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5126655925760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28308381 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚324638451591168 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5285229089280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 _runtime‚2735.248139502 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 173056512 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95621960 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42423799 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚413829102535168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42424090 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14115418 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 trainer/global_step‚5349 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚404423257354080 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54027790 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122677248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54017633 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚408702088297472 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚404423274393440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5117768757342 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95611801 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14115707 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 147101696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41594170 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95621960 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 % _timestamp‚1.7844892586729205e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95611801 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54017633 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490°-ix2¾- _wandbruntime‚2735  _runtime‚2735Î}6x ¿„ 3loss/FBt2/assignment_loss‚0.0027718311968203896 /loss/FRt1/detection_loss‚1.4272005538390786 /loss/FRt2/detection_loss‚0.5402125962777604 1loss/SRqqt2/detection_loss‚1.1367467668410527 /loss/FBt2/detection_loss‚0.6192331237511502 % _timestamp‚1.7844892586739223e+09 0loss/FRt2/assignment_loss‚2.1754299713954883 /loss/FBt1/detection_loss‚0.6707330722173748 _runtime‚2735.248653983 _step‚433 0loss/FRt1/assignment_loss‚1.0004026651382447 1loss/SRqqt2/assignment_loss‚0.703376316577604 2loss/SRqqt1/assignment_loss‚0.5644016211027056 3loss/FBt1/assignment_loss‚0.0004943253153704483 1loss/SRqqt1/detection_loss‚0.6193640863330024 %loss/total_loss‚9.460366929985653 epoch‚6 trainer/global_step‚5349ˆãˆLÀ‰ 3loss/FBt2/assignment_loss‚0.0027718311968203896 /loss/FRt1/detection_loss‚1.4272005538390786 /loss/FRt2/detection_loss‚0.5402125962777604 1loss/SRqqt2/detection_loss‚1.1367467668410527 /loss/FBt2/detection_loss‚0.6192331237511502 % _timestamp‚1.7844892586739223e+09 0loss/FRt2/assignment_loss‚2.1754299713954883 /loss/FBt1/detection_loss‚0.6707330722173748 _runtime‚2735.248653983 _step‚433 0loss/FRt1/assignment_loss‚1.0004026651382447 1loss/SRqqt2/assignment_loss‚0.703376316577604 2loss/SRqqt1/assignment_loss‚0.5644016211027056 3loss/FBt1/assignment_loss‚0.0004943253153704483 1loss/SRqqt1/detection_loss‚0.6193640863330024 %loss/total_loss‚9.460366929985653 epoch‚6 trainer/global_step‚5349±‹NJcÁ‚(:Ø ´ÊôÒ¾³ÅË network.sent‚ 7.18281e+06 network.recv‚756107' disk.nvme1c1n1.out‚7607.10009765625 disk.dm-0.in‚ 4.3359375 disk.dm-1.out‚4061.287109375 disk.rbd0.out‚ 0.4697265625# proc.memory.rssMB‚ 2835.75390625) proc.memory.percent‚17.30806827545166% disk.nvme1n1.out‚7607.10009765625 disk.md0.out‚ 75.0859375 cpu‚26.088503375289847 proc.cpu.threads‚17) disk./.usagePercent‚76.64805531715302& disk.nvme0n1.out‚27359.22607421875 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.nvme0c0n1.in‚ 4.3359375( proc.memory.availableMB‚ 493.74609375% disk./.usageGB‚1176.7392692565918 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0" disk.dm-0.out‚30743.9677734375 disk.dm-1.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚27359.22607421875$ memory_percent‚96.98641300201416PtŽ"žÂ‚(:“ ´ÊôÒëð¸Ì gpu.0.gpu‚64.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚157.387' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚52.46233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞWd2Ã- _wandbruntime‚2759  _runtime‚2759£Ò:$ÈÄ ' lr-AdamW/pg1‚0.00020572261630255503 ' lr-AdamW/pg2‚0.00020572261630255503 trainer/global_step‚5399 $ _timestamp‚1.784489282667929e+09 _runtime‚2759.242844202 _step‚434ùOV‡ÍÅÇ trainer/global_step‚5399 $ _timestamp‚1.784489282667929e+09 _runtime‚2759.242844202 _step‚434 ' lr-AdamW/pg1‚0.00020572261630255503 ' lr-AdamW/pg2‚0.00020572261630255503²BÎÏÞ2Æ- _wandbruntime‚2759  _runtime‚2759¶ —x&ÇßO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚408021152434960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96475271 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚412337484982784 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28557352 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5162943675680 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚417509738819072 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14238879 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚332855543719424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚417509738819072 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚412337502501376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5172594622464 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚332855701314048 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚413184079071280 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚435 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 _runtime‚2759.243523383 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5172253836288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 173056512 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚412337502501376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚412337484982784 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚5399 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41961766 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54503360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42795939 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segmenÅ(u mts.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5172594622464 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5332285003776y·T;ê'ÈäO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96475271 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚412337484982784 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28557352 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5162943675680 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚417509738819072 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14238879 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚332855543719424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚417509738819072 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚412337502501376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5172594622464 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚332855701314048 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚413184079071280 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚435 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 _runtime‚2759.243523383 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5172253836288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 173056512 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚412337502501376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚412337484982784 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚5399 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41961766 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54503360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42795939 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5172594622464 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5332285003776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚408021152434960³§bBe2É- _wandbruntime‚2759  _runtime‚2759aAmEî&ÊèM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚413258433376996 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5173524540928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42803777 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41969268 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚332915382803968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚327582292642816 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96492662 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5164211822350 $ _timestamp‚1.784489283040404e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28562433 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚408094561478080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5163871898916 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54513235 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54513235 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 _runtime‚2759.615535787 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41969270 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14241071 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚436 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 173056512 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42803502 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14241344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚332915532002816 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚327582268218368 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚408094578517440 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96492662 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96482503 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54523392 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚412411694270464 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚412411694270464 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5333239360000 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41969268 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5333114585600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚417584860499456 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚417585218811392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96482503 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41969270 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5173183747584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260ɳ[Ïó&ËíM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42803502 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14241344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚332915532002816 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚327582268218368 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚408094578517440 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96492662 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96482503 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54523392 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚412411694270464 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚412411694270464 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5333239360000 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41969268 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5333114585600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚417584860499456 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚417585218811392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96482503 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41969270 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5173183747584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚413258433376996 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5173524540928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42803777 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41969268 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚332915382803968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚327582292642816 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96492662 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5164211822350 $ _timestamp‚1.784489283040404e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28562433 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚408094561478080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5163871898916 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54513235 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54513235 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 _runtime‚2759.615535787 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41969270 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14241071 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚436 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 173056512 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43176576 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚335847512653824 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚330467232937472 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42336868 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14365176 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚421266213400064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚416047073437184 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5209386740688 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42336866 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚416047073437184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5380279716352 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚416901826260288 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55009105 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54998962 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42336866 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚416047090955776 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97335828 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7844893078592625e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5219122444288 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚421265855095296 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚411692456558960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚416047090955776 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚421266213400064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚335847361350656 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚416902183222970 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5219122444288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5218781658112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚411692439519600“,êòê'ØäO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚421266213400064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚416047073437184 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5209386740688 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42336866 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚416047073437184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5380279716352 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚416901826260288 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55009105 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54998962 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42336866 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚416047090955776 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97335828 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7844893078592625e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5219122444288 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚421265855095296 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚411692456558960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚416047090955776 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚421266213400064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚335847361350656 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚416902183222970 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5219122444288 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5218781658112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚411692439519600 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5218781658112 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42336868 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43176576 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚335847512653824 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚330467232937472 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42336868 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14365176 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260·Ô4b(2Ù- _wandbruntime‚2784  _runtime‚2784¬ÙwÆ%ÚéM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42344368 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5220052362752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5219711569408 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28816483 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚421341335087616 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚416121265206272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚97353205 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14367668 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚411765882641440 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55018994 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚330526109270016 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5381242870784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚97353205 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 _runtime‚2784.805966732 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚421340976775680 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚416121282724864 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5210654887358 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 % _timestamp‚1.7844893082300239e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚335907352140800 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43184438 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚5449 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42344368 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55008837 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14367955 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚335907198747648 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚421341335087616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97363364 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚416121282724864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42344370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚416976180566004 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43184149 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚411765865602080 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5219711569408 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14þªjÑ)407434240 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97363364 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚416121282724864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42344370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚416976180566004 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43184149 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚411765865602080 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5219711569408 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28816483 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚421341335087616 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚416121265206272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚97353205 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14367668 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚411765882641440¸H­É÷2Ü- _wandbruntime‚2784  _runtime‚2784ñV.Ýþ 2loss/SRqqt2/assignment_loss‚0.5940446470215518 /loss/FRt2/detection_loss‚0.5999046822977753 trainer/global_step‚5449 /loss/FRt2/assignment_loss‚2.100881124471689 2loss/FBt1/assignment_loss‚0.001713330757371006 /loss/FRt1/detection_loss‚1.3632648864473131 epoch‚6 _runtime‚2784.806552373 _step‚441 0loss/SRqqt2/detection_loss‚1.160678149181019 /loss/FBt1/detection_loss‚0.3511045532196921 /loss/FBt2/detection_loss‚0.9881350049108732 %loss/total_loss‚9.369712907427985 % _timestamp‚1.7844893082311885e+09 2loss/SRqqt1/assignment_loss‚0.5552689044660656 0loss/FBt2/assignment_loss‚0.0852659867510468 0loss/SRqqt1/detection_loss‚0.648721923379926 0loss/FRt1/assignment_loss‚0.9207297145236623G¬2! Þƒ /loss/FRt2/assignment_loss‚2.100881124471689 2loss/FBt1/assignment_loss‚0.001713330757371006 /loss/FRt1/detection_loss‚1.3632648864473131 epoch‚6 _runtime‚2784.806552373 _step‚441 0loss/SRqqt2/detection_loss‚1.160678149181019 /loss/FBt1/detection_loss‚0.3511045532196921 /loss/FBt2/detection_loss‚0.9881350049108732 %loss/total_loss‚9.369712907427985 % _timestamp‚1.7844893082311885e+09 2loss/SRqqt1/assignment_loss‚0.5552689044660656 0loss/FBt2/assignment_loss‚0.0852659867510468 0loss/SRqqt1/detection_loss‚0.648721923379926 0loss/FRt1/assignment_loss‚0.9207297145236623 2loss/SRqqt2/assignment_loss‚0.5940446470215518 /loss/FRt2/detection_loss‚0.5999046822977753 trainer/global_step‚5449¹’˜ SRß‚(:Ç áÊôÒîáùË" disk.nvme0c0n1.out‚ 27365.78125 disk.md0.out‚75.96875 disk.rbd0.out‚ 0.4697265625# proc.memory.rssMB‚ 2835.75390625' disk./.usagePercent‚76.648054820205 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚7609.20068359375 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚30750.26416015625 disk.dm-1.in‚0# memory_percent‚96.9902753829956 network.sent‚ 7.324991e+06 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.out‚ 27365.78125 disk.md0.in‚0 disk.dm-1.out‚ 4061.71484375) proc.memory.percent‚17.30806827545166 cpu‚26.08805578261633 proc.cpu.threads‚17% disk./.usageGB‚1176.7392616271973 disk.nvme1n1.in‚0% disk.nvme1n1.out‚7609.20068359375 disk.rbd0.in‚0( proc.memory.availableMB‚ 493.11328125 network.recv‚767393yžà‚(:“ áÊôÒ›ÿ±Ì gpu.0.gpu‚81.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚248.402' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚82.80066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…†£cá‚(:Ø ðÊôÒæ´Ë( proc.memory.availableMB‚ 492.85546875 disk.nvme1c1n1.in‚0 disk.md0.out‚ 76.45703125 disk.rbd0.in‚0) proc.memory.percent‚17.30806827545166) disk./.usagePercent‚76.64804985072476 disk.md0.in‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 4.3359375' disk.nvme0c0n1.out‚27367.8974609375% disk.nvme0n1.out‚27367.8974609375$ memory_percent‚96.99184894561768# proc.memory.rssMB‚ 2835.75390625 network.sent‚ 7.373476e+06 network.recv‚771803$ disk./.usageGB‚1176.739185333252 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚30752.06103515625 cpu‚26.086267553564785 proc.cpu.threads‚17& disk.nvme1c1n1.out‚7610.0751953125 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚7610.0751953125" disk.dm-1.out‚4061.87158203125 disk.rbd0.out‚ 0.4697265625 disk.nvme0n1.in‚ 4.3359375…â‚(:“ ðÊôÒ„¾‡Ì gpu.0.gpu‚61.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚251.812' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚83.93733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xÅøÌ2ã- _wandbruntime‚2809  _runtime‚2809 Wü"Éäà ' lr-AdamW/pg1‚0.00020541295680327282 ' lr-AdamW/pg2‚0.00020541295680327282 trainer/global_step‚5499 % _timestamp‚1.7844893325611732e+09 _runtime‚2809.136147984 _step‚442€Ëp+ÎåÈ % _timestamp‚1.7844893325611732e+09 _runtime‚2809.136147984 _step‚442 ' lr-AdamW/pg1‚0.00020541295680327282 ' lr-AdamW/pg2‚0.00020541295680327282 trainer/global_step‚5499ºä\S2æ- _wandbruntime‚2809  _runtime‚2809Á¾z©å'çßO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42711966 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5428312477184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 _step‚443 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚5499 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43557550 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚419756661891584 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98216675 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55504707 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚419756661891584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5428181404160 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55494564 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29065452 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42711968 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42711966 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚425021971371520 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5265650266112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29065450 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43557848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚415363743643600 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55504707 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚419756661891584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5428181404160 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55494564 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29065452 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42711968 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42711966 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚425021971371520 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5265650266112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29065450 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 UGDeviceStatsMonitor.on_train_batch_star›ƒEq€ t/allocated_bytes.small_pool.peak‚ 538870784 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43557848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚415363743643600 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98234066 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚420693927755012 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5256758028932 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚425097451363840 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚419830871179264 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚305 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98223907 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚419830871179264 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42719470 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚425097451363840 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚338899042057216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5266239391232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚333469901472768 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55514596 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚303 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43565414 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29070531 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚420694284717806 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5266580184576 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚425097093051904 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98234066 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55504439 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5429273747456 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚415437186765440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚419830853660672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55514596 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7844893332491186e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚333469925897216 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5266580184576 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133163008 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚415437169726080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5257097952366 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55504439 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43565109 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14494578 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5266239391232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 _runtime‚2809.824279529 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42719468 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664õ@ô&ëîM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55504439 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5429273747456 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚415437186765440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚419830853660672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55514596 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7844893332491186e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚333469925897216 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5266580184576 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133163008 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚415437169726080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5257097952366 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55504439 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43565109 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14494578 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5266239391232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 _runtime‚2809.824279529 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42719468 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42719468 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 trainer/global_step‚5499 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29070533 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚338899199644672 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5429140584448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 173056512 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98234066 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚420693927755012 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5256758028932 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚425097451363840 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚419830871179264 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚305 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98223907 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚419830871179264 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42719470 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚425097451363840 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚338899042057216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5266239391232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚333469901472768 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55514596 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚303 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43565414 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29070531 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚420694284717806 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5266580184576 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚425097093051904 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98234066 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935¼iuw2ì- _wandbruntime‚2809  _runtime‚2809Sˆ ƒí€ 0loss/FRt2/assignment_loss‚2.0518460088027153 /loss/FRt1/detection_loss‚1.5743721082118776 1loss/SRqqt2/detection_loss‚0.9030421052672221 /loss/FBt1/detection_loss‚0.5441236510295185 0loss/FRt1/assignment_loss‚0.8763895729266176 2loss/SRqqt1/assignment_loss‚0.6013802933085108 2loss/FBt1/assignment_loss‚0.000891002883702959 _step‚445 1loss/FBt2/assignment_loss‚0.01362412662052116 /loss/FRt2/detection_loss‚0.5743262630576991 1loss/SRqqt1/detection_loss‚0.7075687465872158 /loss/FBt2/detection_loss‚0.6630760216050677 trainer/global_step‚5499 1loss/SRqqt2/assignment_loss‚0.656171925791691 $loss/total_loss‚9.16681182609236 epoch‚6 % _timestamp‚1.7844893332502058e+09 _runtime‚2809.825368781w{Ÿ$ î… 1loss/FBt2/assignment_loss‚0.01362412662052116 /loss/FRt2/detection_loss‚0.5743262630576991 1loss/SRqqt1/detection_loss‚0.7075687465872158 /loss/FBt2/detection_loss‚0.6630760216050677 trainer/global_step‚5499 1loss/SRqqt2/assignment_loss‚0.656171925791691 $loss/total_loss‚9.16681182609236 epoch‚6 % _timestamp‚1.7844893332502058e+09 _runtime‚2809.825368781 0loss/FRt2/assignment_loss‚2.0518460088027153 /loss/FRt1/detection_loss‚1.5743721082118776 1loss/SRqqt2/detection_loss‚0.9030421052672221 /loss/FBt1/detection_loss‚0.5441236510295185 0loss/FRt1/assignment_loss‚0.8763895729266176 2loss/SRqqt1/assignment_loss‚0.6013802933085108 2loss/FBt1/assignment_loss‚0.000891002883702959 _step‚445½ô¤ŽI\ï‚(:Ñ ÿÊôÒ„áÖË) proc.memory.percent‚17.30806827545166 cpu‚26.08457059360554 network.recv‚774940) disk./.usagePercent‚76.64805382630895% disk./.usageGB‚1176.7392463684082 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 proc.cpu.threads‚17 disk.nvme1n1.in‚0# disk.nvme1n1.out‚7616.994140625 disk.rbd0.out‚ 0.4697265625% disk.nvme0c0n1.out‚27368.83203125% disk.nvme1c1n1.out‚7616.994140625$ memory_percent‚96.99056148529053 network.sent‚ 7.379351e+06 disk.md0.in‚0 disk.md0.out‚ 76.8515625 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚30758.98291015625 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0" disk.dm-1.out‚4061.95751953125# disk.nvme0n1.out‚27368.83203125( proc.memory.availableMB‚ 493.06640625# proc.memory.rssMB‚ 2835.75390625áoøžð‚(:“ ÿÊôÒ«òŒÌ gpu.0.gpu‚62.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚252.125' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.04166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RêU 2ñ- _wandbruntime‚2834  _runtime‚2834<›·ºÇòÁ & lr-AdamW/pg2‚0.0002052557475588404 trainer/global_step‚5549 % _timestamp‚1.7844893575646422e+09 _runtime‚2834.139601476 _step‚446 & lr-AdamW/pg1‚0.0002052557475588404§]ÓhÌóÆ & lr-AdamW/pg1‚0.0002052557475588404 & lr-AdamW/pg2‚0.0002052557475588404 trainer/global_step‚5549 % _timestamp‚1.7844893575646422e+09 _runtime‚2834.139601476 _step‚446¾ÎsþÖ2ô- _wandbruntime‚2834  _runtime‚2834ò4-—ÕõßO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 173056512 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5302612794026 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5302272870704 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55990166 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚341831100673024 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚419035047767600 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 trainer/global_step‚5549 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚2834.140531168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚428778445952512 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚428778087647744 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 % _timestamp‚1.7844893575656428e+09 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14619229 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚424337320638304 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 TFDeviceStatsMonitor.on_train_batch_start/r¬i§òeserved_bytes.small_pool.peak‚ 545259520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5311837301760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43087068 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99077232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43087066 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5476103408128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5476234481152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚428778445952512 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚423466250345984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43087068 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚428778087647744 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚424337677600986 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5312178087936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29319502 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56000309 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚341830945175552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5311837301760 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43938731 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚423466267864576 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29319500 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675Õ¾ÙÚê'öäO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43938731 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚423466267864576 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29319500 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 173056512 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5302612794026 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5302272870704 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55990166 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚341831100673024 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚419035047767600 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 trainer/global_step‚5549 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚2834.140531168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚428778445952512 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚428778087647744 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 % _timestamp‚1.7844893575656428e+09 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14619229 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚424337320638304 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5311837301760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43087068 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99077232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43087066 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5476103408128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5476234481152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚428778445952512 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚423466250345984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43087068 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚428778087647744 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚424337677600986 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5312178087936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29319502 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56000309 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚341830945175552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5311837301760 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0¿ºgœ2÷- _wandbruntime‚2834  _runtime‚2834¯´Ÿ‹ï&øéM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56000041 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5303541017374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 trainer/global_step‚5549 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚336413742524416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5477068650496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚2834.512458315 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚336413718099968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14621714 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43094570 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚424412031906814 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 _step‚448 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5477193424896 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5312767213056 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99104768 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚424411674944020 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚419108490889440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5313108006400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29324581 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚341890786750464 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99094609 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5313108006400 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚423540442115072 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43094570 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43094568 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43946297 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7844893579372647e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚423540459633664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚341890935949312 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5303201093940 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99104768 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚428853567640064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚428853567640064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99094609 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56000041 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚428853209328128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43094568 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56010198 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43946008 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚423540442115072 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29324583 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 173056512 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5312767213056 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0¤4$Â"ùîM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43094570 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚424412031906814 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 _step‚448 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5477193424896 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5312767213056 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99104768 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚424411674944020 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚419108490889440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5313108006400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29324581 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚341890786750464 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99094609 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5313108006400 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚423540442115072 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43094570 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43094568 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43946297 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7844893579372647e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚423540459633664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚341890935949312 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5303201093940 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99104768 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚428853567640064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚428853567640064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99094609 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56000041 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚428853209328128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43094568 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56010198 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43946008 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚423540442115072 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29324583 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 173056512 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5312767213056 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56000041 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5303541017374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 trainer/global||Y@2_step‚5549 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚336413742524416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5477068650496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚2834.512458315 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚336413718099968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14621714 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675À8ð+â2ú- _wandbruntime‚2834  _runtime‚2834r èÕû‚ 2loss/SRqqt1/assignment_loss‚0.6191041476264432 1loss/FBt2/assignment_loss‚0.09649890695481417 % _timestamp‚1.7844893579383404e+09 _runtime‚2834.513009166 _step‚449 /loss/FBt1/detection_loss‚0.4506774914140503 %loss/total_loss‚8.958923755935928 0loss/FRt2/assignment_loss‚2.0921070731821514 /loss/FRt2/detection_loss‚0.5588273695383188 1loss/SRqqt1/detection_loss‚0.7030316550353104 epoch‚6 trainer/global_step‚5549 /loss/FBt2/detection_loss‚0.6551673555001617 0loss/FRt1/assignment_loss‚0.8985340482042742 2loss/SRqqt2/assignment_loss‚0.5916726787240866 3loss/FBt1/assignment_loss‚0.0017508277231350824 /loss/FRt1/detection_loss‚1.3573855315046313 0loss/SRqqt2/detection_loss‚0.934166670528551úWÊ ü‡ 2loss/SRqqt1/assignment_loss‚0.6191041476264432 1loss/FBt2/assignment_loss‚0.09649890695481417 % _timestamp‚1.7844893579383404e+09 _runtime‚2834.513009166 _step‚449 /loss/FBt1/detection_loss‚0.4506774914140503 %loss/total_loss‚8.958923755935928 0loss/FRt2/assignment_loss‚2.0921070731821514 /loss/FRt2/detection_loss‚0.5588273695383188 1loss/SRqqt1/detection_loss‚0.7030316550353104 epoch‚6 trainer/global_step‚5549 /loss/FBt2/detection_loss‚0.6551673555001617 0loss/FRt1/assignment_loss‚0.8985340482042742 2loss/SRqqt2/assignment_loss‚0.5916726787240866 3loss/FBt1/assignment_loss‚0.0017508277231350824 /loss/FRt1/detection_loss‚1.3573855315046313 0loss/SRqqt2/detection_loss‚0.934166670528551Á*k§ˆcý‚(:Ø ŽËôÒγŒÌ) disk./.usagePercent‚76.64721895362942 disk.nvme1n1.in‚0 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0& disk.nvme0n1.out‚27370.89599609375) proc.memory.percent‚17.30806827545166% disk./.usageGB‚1176.7264289855957' disk.nvme1c1n1.out‚7617.82568359375! disk.dm-1.out‚4062.1318359375 disk.rbd0.in‚0$ memory_percent‚96.99487686157227 proc.cpu.threads‚17 network.sent‚ 7.453425e+06 network.recv‚779501 disk.md0.in‚0" disk.dm-0.out‚30760.9365234375 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375& proc.memory.availableMB‚ 492.359375# proc.memory.rssMB‚ 2835.75390625 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚7617.82568359375 disk.md0.out‚ 77.20703125( disk.nvme0c0n1.out‚27370.89599609375 disk.nvme0n1.in‚ 4.3359375 cpu‚26.08297654239145™÷; žþ‚(:“ ŽËôÒÑÒÃÌ gpu.0.gpu‚63.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚256.546' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.51533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!«˜aÿ‚(:Ö ËôÒÁ¾ÖË network.recv‚783656 disk.md0.in‚0 disk.rbd0.out‚ 0.4697265625& disk.nvme0n1.out‚27371.90771484375# proc.memory.rssMB‚ 2835.75390625) proc.memory.percent‚17.30806827545166 cpu‚26.081327831503607 network.sent‚ 7.504821e+06 disk.md0.out‚77.65625 disk.dm-0.in‚ 4.3359375" disk.dm-0.out‚30761.5615234375 disk.dm-1.in‚0( disk.nvme0c0n1.out‚27371.90771484375& proc.memory.availableMB‚ 492.796875' disk.nvme1c1n1.out‚7618.54052734375% disk.nvme1n1.out‚7618.54052734375! disk.dm-1.out‚4062.2724609375 proc.cpu.threads‚17) disk./.usagePercent‚76.64722417158367% disk./.usageGB‚1176.7265090942383 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375$ memory_percent‚96.99220657348633´Ñ˜ž€‚(:“ ËôÒ½Ì‰Ì gpu.0.gpu‚90.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚52.0 gpu.0.powerWatts‚118.674' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.55800000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—ÝZ2- _wandbruntime‚2859  _runtime‚2859Öá*Ç‚Á ' lr-AdamW/pg1‚0.00020509695533344903 ' lr-AdamW/pg2‚0.00020509695533344903 trainer/global_step‚5599 # _timestamp‚1.78448938260879e+09 _runtime‚2859.185626348 _step‚450.Y«ÇÌƒÆ trainer/global_step‚5599 # _timestamp‚1.78448938260879e+09 _runtime‚2859.185626348 _step‚450 ' lr-AdamW/pg1‚0.00020509695533344903 ' lr-AdamW/pg2‚0.00020509695533344903 ‡v…2„- _wandbruntime‚2859  _runtime‚2859Å¡ÿÒä'…ÞO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44319332 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5358365123584 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚432534562228736 % _timestamp‚1.7844893826117804e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5358705909760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5524192998912 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚427175856318976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14746070 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124781568 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚422706368930960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43462168 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56495911 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚344822726611968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚339298682819072 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚ 2859.18677001 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43462166 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚427175838800384 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43462168 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5348715935712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚428055424789994 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5349055859034 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43462166 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5358365123584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14745782 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56485768 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚427175856318976 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚344822875817984 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚432534203923968 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚428055067827312 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚432534562228736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29573552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29573550 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚427175838800384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5524068217344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚339298658394624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 173056512 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5358705909760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56485768 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675-Çbé'†ãO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5349055859034 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43462166 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5358365123584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14745782 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56485768 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚427175856318976 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚344822875817984 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚432534203923968 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚428055067827312 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚432534562228736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29573552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29573550 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚427175838800384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5524068217344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚339298658394624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 173056512 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5358705909760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56485768 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44319332 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5358365123584 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚432534562228736 % _timestamp‚1.7844893826117804e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5358705909760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5524192998912 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚427175856318976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14746070 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124781568 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚422706368930960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43462168 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56495911 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚344822726611968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚339298682819072 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚ 2859.18677001 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43462166 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚427175838800384 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43462168 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5348715935712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚428055424789994ÃðïÏ2‡- _wandbruntime‚2859  _runtime‚2859…V !ˆèM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚427250030569472 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚427250048088064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5359635828224 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56495643 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29578633 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56505800 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99965311 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5525031091200 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5359295034880 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99965311 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14748606 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 _runtime‚2859.565663696 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44327239 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚427250048088064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚297 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44326940 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚427250030569472 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5359295034880 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 147101696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122677248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99975470 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚422779777974080 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99975470 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 $ _timestamp‚1.784489382990471e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43469670 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚428129422133028 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚344882565818368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43469668 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 \HDeviceStatsMonitor.on_train_baÃv#Í tch_end/inactive_split_bytes.all.allocated‚344882712920064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 173056512 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56505800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚299 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚339357534727168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚428129779095822 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5349644158948 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚432609683916288 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚432609683916288 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚339357559151616 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43469668 trainer/global_step‚5599 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56495643 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5359635828224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14748309 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5525153768448µó&‰íM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 $ _timestamp‚1.784489382990471e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43469670 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚428129422133028 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚344882565818368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43469668 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚344882712920064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 173056512 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56505800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚299 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚339357534727168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚428129779095822 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5349644158948 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚432609683916288 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚432609683916288 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚339357559151616 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43469668 trainer/global_step‚5599 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56495643 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5359635828224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14748309 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5525153768448 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚427250030569472 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚427250048088064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5359635828224 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56495643 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29578633 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56505800 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99965311 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5525031091200 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5359295034880 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99965311 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14748606 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 _runtime‚2859.565663696 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44327239 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚427250048088064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚297 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44326940 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚427250030569472 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5359295034880 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 147101696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122677248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99975470 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚422779777974080 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99975470 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446Äý£±2Š- _wandbruntime‚2859  _runtime‚2859Ö[‹‹ü /loss/FRt1/assignment_loss‚1.055320042567175 2loss/SRqqt2/assignment_loss‚0.5927907245687062 /loss/FRt1/detection_loss‚1.3815646080231507 -loss/FRt2/detection_loss‚0.55490365846778 1loss/SRqqt2/detection_loss‚0.9727370948163107 epoch‚6 1loss/FBt1/assignment_loss‚0.07282752119808797 .loss/FBt1/detection_loss‚0.590368689175543 2loss/SRqqt1/assignment_loss‚0.6073154551052052 1loss/FBt2/assignment_loss‚0.02827134993332541 1loss/SRqqt1/detection_loss‚0.6839137101821152 %loss/total_loss‚9.368736846914512 trainer/global_step‚5599 % _timestamp‚1.7844893829922419e+09 _step‚453 /loss/FRt2/assignment_loss‚2.065173769122676 /loss/FBt2/detection_loss‚0.7635502237544367 _runtime‚2859.567057709~vÝ‚Œ _step‚453 /loss/FRt2/assignment_loss‚2.065173769122676 /loss/FBt2/detection_loss‚0.7635502237544367 _runtime‚2859.567057709 /loss/FRt1/assignment_loss‚1.055320042567175 2loss/SRqqt2/assignment_loss‚0.5927907245687062 /loss/FRt1/detection_loss‚1.3815646080231507 -loss/FRt2/detection_loss‚0.55490365846778 1loss/SRqqt2/detection_loss‚0.9727370948163107 epoch‚6 1loss/FBt1/assignment_loss‚0.07282752119808797 .loss/FBt1/detection_loss‚0.590368689175543 2loss/SRqqt1/assignment_loss‚0.6073154551052052 1loss/FBt2/assignment_loss‚0.02827134993332541 1loss/SRqqt1/detection_loss‚0.6839137101821152 %loss/total_loss‚9.368736846914512 trainer/global_step‚5599 % _timestamp‚1.7844893829922419e+09Å+®Y›‚(: ¬ËôÒþ½Ë gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚262.09' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.36333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ:Í_Ž‚(:Ô ¬ËôÒDZÁÌ( disk./.usagePercent‚76.6472249170057$ disk.nvme1n1.out‚7619.6201171875 disk.md0.in‚0( disk.nvme0c0n1.out‚27375.46435546875 disk.nvme0n1.in‚ 4.3359375( proc.memory.availableMB‚ 492.26953125) proc.memory.percent‚17.31107234954834 network.sent‚ 7.520581e+06 network.recv‚787003 disk.md0.out‚ 78.1015625" disk.dm-0.out‚30765.1337890625 disk.dm-1.in‚0" disk.dm-1.out‚4062.39111328125 disk.rbd0.in‚0# disk./.usageGB‚1176.72652053833 disk.dm-0.in‚ 4.3359375 disk.rbd0.out‚ 0.4697265625& disk.nvme0n1.out‚27375.46435546875 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.nvme0c0n1.in‚ 4.3359375& disk.nvme1c1n1.out‚7619.6201171875# memory_percent‚96.9954252243042# proc.memory.rssMB‚ 2836.24609375 cpu‚26.08184341979754 proc.cpu.threads‚17N^‚(:Ó »ËôÒ‚ŽËË& disk.nvme0n1.out‚27376.56982421875# memory_percent‚97.0050573348999* proc.memory.percent‚17.320632934570312) disk./.usagePercent‚76.64725572778316$ disk./.usageGB‚1176.726993560791 disk.nvme1c1n1.in‚0 disk.md0.out‚ 78.3203125" disk.dm-0.out‚30766.2275390625 disk.rbd0.out‚ 0.4697265625 disk.nvme0n1.in‚ 4.3359375( proc.memory.availableMB‚ 490.69140625 network.recv‚791182$ disk.nvme1n1.out‚7620.1630859375 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0 proc.memory.rssMB‚ 2837.8125 cpu‚26.09577710575198 proc.cpu.threads‚17& disk.nvme1c1n1.out‚7620.1630859375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚27376.56982421875 network.sent‚ 7.572055e+06 disk.nvme1n1.in‚0" disk.dm-1.out‚4062.45361328125 disk.nvme0c0n1.in‚ 4.3359375–Å*‚(:’ »ËôÒѱûË gpu.0.gpu‚21.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚264.769' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.25633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p½z2‘- _wandbruntime‚2882  _runtime‚2882‡ž§‘:’{ 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17434673011302948 particle/accuracy_5_of_5‚0 (loss/val_total_loss‚8.71963278059708 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20463888347148895 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6840022802352905 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_4_of_5‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 .particle/accuracy_3_of_3‚0.999514102935791 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8706141114234924 jet/accuracy_1_of_5‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15895655751228333 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7897217273712158 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938561320304871 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6619845032691956 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1577984094619751 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.14034920930862427 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7645201683044434 particle/accuracy_2_of_2‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14834025502204895 +jet/accuracy_3_of_4‚0.09097576141357422 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8709402084350586 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8695986866950989 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18217846751213074 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6398946046829224 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9909990429878235 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8762821555137634 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.992935299873352 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04135959595441818 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1785714328289032 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20075879991054535 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14526240527629852 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9707105755805969 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6302424669265747 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14354971051216125 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6910714507102966 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9825699925422668 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38967373967170715 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7045491933822632 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1682525873184204 (particle/accuracy‚0.8494078516960144 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6882354021072388 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.898996889591217 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5983327031135559 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12932603061199188 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 +jet/accuracy_2_of_2‚0.14900191128253937 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8662281036376953 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7512686848640442 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6562279462814331 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.27272728085517883 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7918803095817566 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.49640020728111267 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.038775186985731125 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3050796687602997 6validation_average_jet_accuracy‚0.3558168411254883 epoch‚6 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.66252201795578 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5085936784744263 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1621745377779007 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9875572919845581 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6193011403083801 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4454926550388336 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7204856276512146 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6419708132743835 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02977650798857212 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 _step‚454 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5480818748474121 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8500000238418579 jet/accuracy_2_of_5‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949186444282532 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998813271522522 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5539828538894653 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.746344268321991 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8794472813606262 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9429416656494141 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6878758668899536 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9124273657798767 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7244400382041931 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7464801669120789 *jet/accuracy_2_of_3‚0.5118159055709839 particle/accuracy_1_of_2‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6651131510734558 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6584351658821106 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.7161628603935242 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9959749579429626 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3703703284263611 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5826342105865479 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02686466835439205 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02977650798857212 jet/accuracy_3_of_5‚1 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.42424246668815613 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02530190721154213 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.013942567631602287 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12794454395771027 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6407735347747803 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9993197321891785 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 +particle/sensitivity‚0.6323696374893188 particle/accuracy_2_of_5‚1 _runtime‚2882.309626045 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1349572241306305 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9125000238418579 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2777777910232544 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994588494300842 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09936507791280746 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9798571467399597 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7205389738082886 +particle/specificity‚0.9087616205215454 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7192688584327698 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02261609025299549 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6314110159873962 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09280136972665787 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8500000238418579 /particle/accuracy_3_of_4‚0.9965986609458923 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.29727205634117126 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9993197321891785 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7200756669044495 *jet/accuracy_1_of_3‚0.9169816374778748 'particle/f_score‚0.6432722806930542 6 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.569444477558136 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9946548938751221 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6859627962112427 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.45831209421157837 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 $ _timestamp‚1.784489405734157e+09 *jet/accuracy_1_of_2‚0.7604467868804932 particle/accuracy_1_of_3‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38676896691322327 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1563226878643036 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11579481512308121 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7627815008163452 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_3‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6984577178955078 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9962481260299683 +jet/accuracy_1_of_1‚0.31329581141471863 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1682525873184204 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1083393320441246 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18192629516124725 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11562500149011612 particle/accuracy_4_of_5‚1 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.039930835366249084 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_1_of_4‚1 particle/accuracy_2_of_4‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18181996047496796 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4635933041572571 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03823168948292732 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6619845032691956 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 * Purity/0FRt1SRqqt2FBt/FBt_purity‚0.875 )jet/accuracy_1_of_4‚0.946182906627655 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9853552579879761 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7201130986213684 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8881579041481018 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.658554196357727 *jet/accuracy_3_of_3‚0.0750986784696579 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6582088470458984 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7268853187561035 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7413356900215149 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15152308344841003 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16721267998218536 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978741407394409 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03847559168934822 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4274737536907196 ,jet/accuracy_4_of_4‚0.018498167395591736 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.25810080766677856 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.23809780180454254 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9880084991455078 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6651399731636047 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.998134970664978 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03943752497434616 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.78972í²17273712158 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45730364322662354 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1785714328289032 trainer/global_step‚5606 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9429877996444702 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4699999988079071>SOG¨=“¢{ 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9853552579879761 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7201130986213684 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8881579041481018 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.658554196357727 *jet/accuracy_3_of_3‚0.0750986784696579 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6582088470458984 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7268853187561035 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7413356900215149 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15152308344841003 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16721267998218536 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978741407394409 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03847559168934822 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4274737536907196 ,jet/accuracy_4_of_4‚0.018498167395591736 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.25810080766677856 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.23809780180454254 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9880084991455078 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6651399731636047 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.998134970664978 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03943752497434616 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7897217273712158 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45730364322662354 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1785714328289032 trainer/global_step‚5606 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9429877996444702 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4699999988079071 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17434673011302948 particle/accuracy_5_of_5‚0 (loss/val_total_loss‚8.71963278059708 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20463888347148895 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6840022802352905 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_4_of_5‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 .particle/accuracy_3_of_3‚0.999514102935791 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8706141114234924 jet/accuracy_1_of_5‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15895655751228333 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7897217273712158 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938561320304871 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6619845032691956 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1577984094619751 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.14034920930862427 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7645201683044434 particle/accuracy_2_of_2‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14834025502204895 +jet/accuracy_3_of_4‚0.09097576141357422 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8709402084350586 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8695986866950989 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18217846751213074 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6398946046829224 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9909990429878235 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8762821555137634 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.992935299873352 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04135959595441818 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1785714328289032 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20075879991054535 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14526240527629852 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9707105755805969 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6302424669265747 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14354971051216125 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6910714507102966 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9825699925422668 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38967373967170715 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7045491933822632 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1682525873184204 (particle/accuracy‚0.8494078516960144 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6882354021072388 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.898996889591217 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5983327031135559 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12932603061199188 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 +jet/accuracy_2_of_2‚0.14900191128253937 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8662281036376953 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7512686848640442 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6562279462814331 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.27272728085517883 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7918803095817566 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.49640020728111267 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.038775186985731125 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3050796687602997 6validation_average_jet_accuracy‚0.3558168411254883 epoch‚6 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.66252201795578 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5085936784744263 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1621745377779007 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9875572919845581 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6193011403083801 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4454926550388336 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7204856276512146 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6419708132743835 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02977650798857212 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 _step‚454 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5480818748474121 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8500000238418579 jet/accuracy_2_of_5‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949186444282532 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998813271522522 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5539828538894653 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.746344268321991 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8794472813606262 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9429416656494141 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6878758668899536 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9124273657798767 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7244400382041931 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7464801669120789 *jet/accuracy_2_of_3‚0.5118159055709839 particle/accuracy_1_of_2‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6651131510734558 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6584351658821106 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.7161628603935242 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9959749579429626 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3703703284263611 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5826342105865479 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02686466835439205 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02977650798857212 jet/accuracy_3_of_5‚1 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.42424246668815613 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02530190721154213 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.013942567631602287 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12794454395771027 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6407735347747803 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9993197321891785 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 +particle/sensitivity‚0.6323696374893188 particle/accuracy_2_of_5‚1 _runtime‚2882.309626045 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1349572241306305 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9125000238418579 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2777777910232544 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994588494300842 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09936507791280746 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9798571467399597 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7205389738082886 +particle/specificity‚0.9087616205215454 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7192688584327698 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02261609025299549 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6314110159873962 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09280136972665787 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8500000238418579 /particle/accuracy_3_of_4‚0.9965986609458923 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.29727205634117126 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9993197321891785 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7200756669044495 *jet/accuracy_1_of_3‚0.9169816374778748 'particle/f_score‚0.6432722806930542 6 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.569444477558136 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9946548938751221 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6859627962112427 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.45831209421157837 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 $ _timestamp‚1.784489405734157e+09 *jet/accuracy_1_of_2‚0.7604467868804932 particle/accuracy_1_of_3‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38676896691322327 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1563226878643036 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11579481512308121 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7627815008163452 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_3‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6984577178955078 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9962481260299683 +jet/accuracy_1_of_1‚0.31329581141471863 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1682525873184204 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1083393320441246 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18192629516124725 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11562500149011612 particle/accuracy_4_of_5‚1 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.039930835366249084 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_1_of_4‚1 particle/accuracy_2_of_4‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18181996047496796 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4635933041572571 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03823168948292732 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6619845032691956 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 * Purity/0FRt1SRqqt2FBt/FBt_purity‚0.875 )jet/accuracy_1_of_4‚0.946182906627655 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064ÆpÛÞ `”‚(:Õ ÊËôÒÃÔË& disk.nvme1c1n1.out‚7866.3251953125 disk.dm-1.in‚0 network.recv‚794928 disk.nvme0n1.in‚ 4.3359375 disk.md0.in‚0 disk.md0.out‚ 78.86328125" disk.dm-1.out‚4307.91357421875 cpu‚26.100782270706947) disk./.usagePercent‚76.64726442437357 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚7866.3251953125# disk.dm-0.out‚30767.47216796875$ memory_percent‚95.49837112426758 proc.cpu.threads‚17 disk.rbd0.in‚0' disk.nvme0c0n1.out‚27378.2744140625 disk.nvme1c1n1.in‚0 disk.dm-0.in‚ 4.3359375& proc.memory.availableMB‚ 737.546875! proc.memory.rssMB‚ 2841.171875* proc.memory.percent‚17.341136932373047 network.sent‚ 7.57512e+06% disk./.usageGB‚1176.7271270751953% disk.nvme0n1.out‚27378.2744140625+žGž•‚(:“ ÊËôÒÖ˵Ì gpu.0.gpu‚90.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚51.0 gpu.0.powerWatts‚256.223' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.40766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F6ä2–- _wandbruntime‚2905  _runtime‚2905ÑÜ-5È—Â ' lr-AdamW/pg1‚0.00020493658267047848 ' lr-AdamW/pg2‚0.00020493658267047848 trainer/global_step‚5649 $ _timestamp‚1.784489428877763e+09 _runtime‚2905.452824232 _step‚455YÅ–bÍ˜Ç ' lr-AdamW/pg1‚0.00020493658267047848 ' lr-AdamW/pg2‚0.00020493658267047848 trainer/global_step‚5649 $ _timestamp‚1.784489428877763e+09 _runtime‚2905.452824232 _step‚455Ç££»_2™- _wandbruntime‚2905  _runtime‚2905öõâé'šãO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚432832417908224 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5429409000448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57144940 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚433714376183630 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57155083 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚438262185213440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚438261826908672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5429749786624 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44078430 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚349432218655232 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5419988927512 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5429749786624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 _runtime‚2905.462468218 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44078432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 _step‚456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚428294744218800 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5419649004190 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5596857541632 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚428294727179440 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 trainer/global_step‚5649 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57144940 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚349432367861248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44078430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚432832435426816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30014080 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44979997 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30014082 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5596982323200 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚433714733146312 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚438262185213440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44980270 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚343835385538048 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101223370 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14965917 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚343835361113600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7844894288841295e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14966188 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5429409000448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚438261826908672 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚432832435426816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 trainer/global_step‚5649 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57144940 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚349432367861248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44078430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚432832435426816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30014080 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44979997 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 XKDeviceSt4‚•ÎatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30014082 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5596982323200 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚433714733146312 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚438262185213440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44980270 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚343835385538048 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101223370 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14965917 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚343835361113600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7844894288841295e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14966188 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5429409000448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚438261826908672 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚432832435426816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280È?:€2œ- _wandbruntime‚2905  _runtime‚2905Å 0ò&ìM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5430679705088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 $ _timestamp‚1.784489429256284e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30019163 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5430338911744 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57154815 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30019161 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44987537 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚349492208564736 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44085932 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚349492057268736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚433789087452140 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44085932 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101240747 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57164972 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5597819822592 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57154815 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚438337306900992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚432906609677312 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚343894261870592 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5597946694144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚438336948589056 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44987825 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚5649 _step‚457 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚343894237446144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5420577227426 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44085934 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14968662 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101250906 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚433788730489346 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101250906 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚428368170301280 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚432906627195904 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44085934 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 _runtime‚2905.831479644 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚438337306900992 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚343894237446144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5420577227426 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44085934 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14968662 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101250906 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚433788730489346 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101250906 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚428368170301280 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚432906627195904 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44085934 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 _runtime‚2905.831479644 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚438337306900992 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 $ _timestamp‚1.784489429256284e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30019163 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5430338911744 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57154815 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30019161 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44987537 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚349492208564736 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520ÉKM‰Ê2Ÿ- _wandbruntime‚2905  _runtime‚2905Óïå  ƒ _runtime‚2905.832826676 /loss/FBt2/detection_loss‚0.7073529312759638 epoch‚7 2loss/SRqqt1/assignment_loss‚0.6704451757190139 2loss/SRqqt2/assignment_loss‚0.5417188501755875 1loss/SRqqt1/detection_loss‚0.6716853427043687 /loss/FRt2/assignment_loss‚2.090812667672123 /loss/FRt2/detection_loss‚0.4784201468656712 %loss/total_loss‚8.897622447642295 trainer/global_step‚5649 _step‚458 0loss/FRt1/assignment_loss‚1.0697530880819595 4loss/FBt1/assignment_loss‚0.00028875198521051616 /loss/FRt1/detection_loss‚1.2877851270270038 1loss/SRqqt2/detection_loss‚0.9282289188411462 /loss/FBt1/detection_loss‚0.3960799720842347 $ _timestamp‚1.784489429258051e+09 2loss/FBt2/assignment_loss‚0.055051475210011344mö2-¡ˆ 1loss/SRqqt1/detection_loss‚0.6716853427043687 /loss/FRt2/assignment_loss‚2.090812667672123 /loss/FRt2/detection_loss‚0.4784201468656712 %loss/total_loss‚8.897622447642295 trainer/global_step‚5649 _step‚458 0loss/FRt1/assignment_loss‚1.0697530880819595 4loss/FBt1/assignment_loss‚0.00028875198521051616 /loss/FRt1/detection_loss‚1.2877851270270038 1loss/SRqqt2/detection_loss‚0.9282289188411462 /loss/FBt1/detection_loss‚0.3960799720842347 $ _timestamp‚1.784489429258051e+09 2loss/FBt2/assignment_loss‚0.055051475210011344 _runtime‚2905.832826676 /loss/FBt2/detection_loss‚0.7073529312759638 epoch‚7 2loss/SRqqt1/assignment_loss‚0.6704451757190139 2loss/SRqqt2/assignment_loss‚0.5417188501755875Êà5à+ž¢‚(:“ ÙËôÒ›»…Ì gpu.0.gpu‚54.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚52.0 gpu.0.powerWatts‚259.935' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.64500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~va£‚(:Ö ÙËôÒºÁ‰Ï! proc.memory.rssMB‚ 2841.171875) disk./.usagePercent‚76.64731312527988 disk.nvme0c0n1.in‚ 4.3359375& disk.nvme1c1n1.out‚7867.0068359375 disk.md0.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 network.sent‚ 7.646001e+06 network.recv‚798767% disk./.usageGB‚1176.7278747558594% disk.nvme0n1.out‚27379.4482421875 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚7867.0068359375 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0' disk.nvme0c0n1.out‚27379.4482421875 disk.nvme0n1.in‚ 4.3359375 disk.md0.out‚ 79.1953125( proc.memory.availableMB‚ 737.25390625* proc.memory.percent‚17.341136932373047 cpu‚26.10396360052771 disk.nvme1c1n1.in‚0# disk.dm-0.out‚30768.56201171875" disk.dm-1.out‚4307.95654296875 disk.rbd0.out‚ 0.4697265625$ memory_percent‚95.50015926361084“(›ê]¤‚(:Ò èËôÒí¬ÉË disk.rbd0.in‚0 proc.cpu.threads‚17 network.sent‚ 7.694175e+06% disk./.usageGB‚1174.0729637145996 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚8119.64404296875! proc.memory.rssMB‚ 2841.171875* proc.memory.percent‚17.341136932373047 cpu‚26.107155789295774 network.recv‚802157 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0 disk.dm-1.out‚ 4553.51171875 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1n1.in‚0" proc.memory.availableMB‚737.25) disk./.usagePercent‚76.47438291578568 disk.md0.out‚ 79.75390625" disk.dm-0.out‚30778.5693359375 disk.rbd0.out‚ 0.4697265625( disk.nvme0c0n1.out‚27383.57666015625& disk.nvme0n1.out‚27383.57666015625' disk.nvme1c1n1.out‚8119.64404296875$ memory_percent‚95.50018310546875sG­“¥‚(:ˆ èËôÒ‚±¹Ë gpu.0.gpu‚53.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚244.584' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚81.528 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à“42¦- _wandbruntime‚2931  _runtime‚2931^yLɧà ' lr-AdamW/pg1‚0.00020477463213862244 ' lr-AdamW/pg2‚0.00020477463213862244 trainer/global_step‚5699 % _timestamp‚1.7844894547311344e+09 _runtime‚2931.306687896 _step‚459Ö©Î¨È % _timestamp‚1.7844894547311344e+09 _runtime‚2931.306687896 _step‚459 ' lr-AdamW/pg1‚0.00020477463213862244 ' lr-AdamW/pg2‚0.00020477463213862244 trainer/global_step‚5699Ë?¹42©- _wandbruntime‚2931  _runtime‚2931ÿ(átªâO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57640542 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚442018301489664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30268132 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚295 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5644832703488 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚431966031303440 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5476277608448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15091999 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚437432123372638 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5476277608448 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚442017943184896 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45360131 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚ 2931.30864099 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45359834 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚346779202165248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5466431992520 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚442018301489664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚442017943184896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44453532 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚431966048342800 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚436542006362624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚436542006362624 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚436542023881216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚297 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 trainer/global_step‚5699 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5466092069198 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5476277608448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15091999 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚437432123372638 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5476277608448 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚442017943184896 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45360131 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚ 2931.30864099 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45359834 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚346779202165248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5466431992520 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚442018301489664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚442017943184896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44453532 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚431966048342800 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚436542006362624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚436542006362624 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚436542023881216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚297 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 trainer/global_step‚5699 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5466092069198 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143Ì™¦ë2¬- _wandbruntime‚2931  _runtime‚2931¿‹™ð&­êM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45367334 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚346838078497792 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44461034 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚442093064865280 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45367607 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5645798369792 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 # _timestamp‚1.78448945510943e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102111449 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚432039457385920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102121608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57660574 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5699 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30273213 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30273211 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚436616215650304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5476866733568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57660574 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚436616198131712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15094123 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15094394 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚437506834641148 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44461032 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57650417 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5476866733568 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚346838054073344 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57650417 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44461032 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 _step‚461 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚442093423177216 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚352484003739136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚352483852443136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5645925241344 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675ÑÔ_Z®ïM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45367607 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5645798369792 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 # _timestamp‚1.78448945510943e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102111449 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚432039457385920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102121608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57660574 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5699 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30273213 JDDeviceStatsMonitor.on_train_batchCÂï›_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30273211 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚436616215650304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5476866733568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57660574 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚436616198131712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15094123 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15094394 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚437506834641148 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44461032 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57650417 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5476866733568 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚346838054073344 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57650417 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44461032 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 _step‚461 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚442093423177216 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚352484003739136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚352483852443136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5645925241344 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45367334 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚346838078497792 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44461034 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚442093064865280ÍíÑ‹¡2¯- _wandbruntime‚2931  _runtime‚2931Á8Yz° 3loss/FBt1/assignment_loss‚0.0005959576540502331 .loss/FRt1/detection_loss‚1.495440202020842 1loss/SRqqt1/detection_loss‚0.6431944616276365 1loss/SRqqt2/detection_loss‚1.0310486893458162 1loss/FBt2/assignment_loss‚0.01598760343365626 .loss/FRt2/detection_loss‚0.557453291220373 0loss/FBt2/detection_loss‚0.47601649998687207 %loss/total_loss‚8.639840085048142 epoch‚7 _runtime‚2931.686439473 _step‚462 2loss/SRqqt1/assignment_loss‚0.5615727815418352 2loss/SRqqt2/assignment_loss‚0.6504825098496619 /loss/FBt1/detection_loss‚0.3152859817307718 0loss/FRt1/assignment_loss‚0.9193376931618518 trainer/global_step‚5699 $ _timestamp‚1.784489455111579e+09 0loss/FRt2/assignment_loss‚1.97342441347477521Ñÿ… ±† 0loss/FRt2/assignment_loss‚1.9734244134747752 3loss/FBt1/assignment_loss‚0.0005959576540502331 .loss/FRt1/detection_loss‚1.495440202020842 1loss/SRqqt1/detection_loss‚0.6431944616276365 1loss/SRqqt2/detection_loss‚1.0310486893458162 1loss/FBt2/assignment_loss‚0.01598760343365626 .loss/FRt2/detection_loss‚0.557453291220373 0loss/FBt2/detection_loss‚0.47601649998687207 %loss/total_loss‚8.639840085048142 epoch‚7 _runtime‚2931.686439473 _step‚462 2loss/SRqqt1/assignment_loss‚0.5615727815418352 2loss/SRqqt2/assignment_loss‚0.6504825098496619 /loss/FBt1/detection_loss‚0.3152859817307718 0loss/FRt1/assignment_loss‚0.9193376931618518 trainer/global_step‚5699 $ _timestamp‚1.784489455111579e+09ÎÔÂ&W²‚(:Ì ÷ËôÒ¾ÅˆÌ network.recv‚804943) disk./.usagePercent‚76.47438540052579% disk./.usageGB‚1174.0730018615723 disk.nvme0c0n1.in‚ 4.3359375' disk.nvme0c0n1.out‚27384.3818359375 disk.md0.out‚80.03125 disk.dm-1.in‚0 cpu‚26.10584079475882 disk.nvme1n1.in‚0" disk.dm-0.out‚30779.1396484375 disk.dm-1.out‚ 4553.55859375$ memory_percent‚95.49901485443115" disk.nvme1n1.out‚ 8120.05859375 network.sent‚ 7.712161e+06 disk.rbd0.in‚0$ disk.nvme1c1n1.out‚ 8120.05859375! proc.memory.rssMB‚ 2841.171875 proc.cpu.threads‚17 disk.rbd0.out‚ 0.4697265625 disk.nvme0n1.in‚ 4.3359375% disk.nvme0n1.out‚27384.3818359375 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375( proc.memory.availableMB‚ 737.44140625* proc.memory.percent‚17.341136932373047&B}éž³‚(:“ ÷ËôÒþã¼Ì gpu.0.gpu‚90.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚52.0 gpu.0.powerWatts‚255.577' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.19233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l‚úž´‚(:“ †ÌôÒ–ÈÆË gpu.0.gpu‚90.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚205.072' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚68.35733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Jï|P]µ‚(:Ò †ÌôÒÒ‰ŸÌ disk.dm-1.in‚0 network.sent‚ 7.762474e+06 network.recv‚809497 disk.nvme1n1.in‚0# disk.dm-0.out‚30815.84912109375$ memory_percent‚95.49899101257324' proc.memory.availableMB‚ 737.4453125! proc.memory.rssMB‚ 2841.171875* proc.memory.percent‚17.341136932373047 disk.nvme1c1n1.in‚0$ disk.nvme1n1.out‚8123.7783203125 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625 disk.md0.in‚0 disk.md0.out‚ 80.390625 cpu‚26.103871297050343 disk.dm-0.in‚ 4.3359375" disk.dm-1.out‚4555.49755859375 proc.cpu.threads‚17& disk.nvme1c1n1.out‚8123.7783203125' disk./.usagePercent‚76.404906848882$ disk./.usageGB‚1173.006332397461 disk.nvme0c0n1.in‚ 4.3359375' disk.nvme0c0n1.out‚27420.0830078125 disk.nvme0n1.in‚ 4.3359375% disk.nvme0n1.out‚27420.0830078125íÞÌÈ2¶- _wandbruntime‚2956  _runtime‚2956;Q ÏÉ·Ã ' lr-AdamW/pg1‚0.00020461110633184743 ' lr-AdamW/pg2‚0.00020461110633184743 trainer/global_step‚5749 % _timestamp‚1.7844894803204365e+09 _runtime‚2956.895423905 _step‚463‚WÝ>Î¸È ' lr-AdamW/pg1‚0.00020461110633184743 ' lr-AdamW/pg2‚0.00020461110633184743 trainer/global_step‚5749 % _timestamp‚1.7844894803204365e+09 _runtime‚2956.895423905 _step‚463ÏrAs2¹- _wandbruntime‚2956  _runtime‚2956ñ!(é'ºãO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚5749 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58146287 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚435637335427440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚355415840897536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚349722994368000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5522805430272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5522464644096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58146287 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44828632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5522805430272 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15218057 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚441150227524328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚440251612335616 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚435637352466800 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5692846529536 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44828630 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45739946 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30522182 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚355415992200704 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5692973408256 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚445774417765888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44828632 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58136144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚445774059461120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5512535134206 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚440251612335616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15217766 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 _step‚464 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 % _timestamp‚1.7844894803215017e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5522464644096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚445774417765888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30522180 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚441149870561646 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102964774 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚445774059461120 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚440251594817024 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚2956.896523407 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935Úú}rî'»èO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚435637335427440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚355415840897536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚349722994368000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5522805430272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5522464644096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58146287 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44828632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5522805430272 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15218057 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚441150227524328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚440251612335616 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚435637352466800 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5692846529536 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44828630 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45739946 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30522182 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚355415992200704 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5692973408256 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚445774417765888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44828632 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58136144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚445774059461120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5512535134206 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚440251612335616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15217766 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 _step‚464 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 % _timestamp‚1.7844894803215017e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5522464644096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚445774417765888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30522180 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚441149870561646 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102964774 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚445774059461120 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚440251594817024 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚2956.896523407 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚5749 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58146287 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ÐDàŠ¾2¼- _wandbruntime‚2957  _runtime‚2957´)ô™¼½ìM FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚441224581830156 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚445849539453440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚349781870700544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚440325804104704 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚355475680156160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 H@DeviceStatsMonitor.on_train_batchtÇÈq6 _end/inactive_split.all.current‚282 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 _runtime‚2957.274915704 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5523735348736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚440325786586112 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5523394555392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5523735348736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚440325786586112 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚435710761509920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45747901 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45747619 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30527263 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15220358 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5693809455616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5693934230016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44836134 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚441224224867362 _step‚465 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚440325804104704 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102992310 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚435710778549280 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102992310 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚441224581830156 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚445849539453440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚349781870700544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚440325804104704 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚355475680156160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 _runtime‚2957.274915704 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5523735348736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚440325786586112 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5523394555392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5523735348736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚440325786586112 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚435710761509920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45747901 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45747619 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30527263 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15220358 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5693809455616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5693934230016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44836134 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚441224224867362 _step‚465 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚440325804104704 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102992310 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚435710778549280 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240Ñ0—ô2¿- _wandbruntime‚2957  _runtime‚2957PêiÀ€ 0loss/FRt1/assignment_loss‚1.0582279707305133 /loss/FBt1/detection_loss‚0.5389468638417644 /loss/FBt2/detection_loss‚0.7238064138868521 trainer/global_step‚5749 /loss/FRt2/assignment_loss‚2.049332031297468 1loss/FBt1/assignment_loss‚0.01790715515692311 1loss/SRqqt2/detection_loss‚1.0914643646045417 %loss/total_loss‚9.233497781675275 epoch‚7 2loss/SRqqt2/assignment_loss‚0.6304845165651015 2loss/FBt2/assignment_loss‚0.020958609332970897 .loss/FRt1/detection_loss‚1.380407926982116 /loss/FRt2/detection_loss‚0.4982464129673138 % _timestamp‚1.7844894807009027e+09 _runtime‚2957.275625495 2loss/SRqqt1/assignment_loss‚0.6123690806789155 1loss/SRqqt1/detection_loss‚0.6113464356307964 _step‚466!B's Á… 2loss/SRqqt2/assignment_loss‚0.6304845165651015 2loss/FBt2/assignment_loss‚0.020958609332970897 .loss/FRt1/detection_loss‚1.380407926982116 /loss/FRt2/detection_loss‚0.4982464129673138 % _timestamp‚1.7844894807009027e+09 _runtime‚2957.275625495 2loss/SRqqt1/assignment_loss‚0.6123690806789155 1loss/SRqqt1/detection_loss‚0.6113464356307964 _step‚466 0loss/FRt1/assignment_loss‚1.0582279707305133 /loss/FBt1/detection_loss‚0.5389468638417644 /loss/FBt2/detection_loss‚0.7238064138868521 trainer/global_step‚5749 /loss/FRt2/assignment_loss‚2.049332031297468 1loss/FBt1/assignment_loss‚0.01790715515692311 1loss/SRqqt2/detection_loss‚1.0914643646045417 %loss/total_loss‚9.233497781675275 epoch‚7ÒCˆÛlc‚(:Ø •ÌôÒæážÌ network.sent‚ 7.778239e+06 disk.rbd0.out‚ 0.4697265625% disk.nvme0n1.out‚27449.1552734375 disk.md0.out‚ 80.78515625# disk.dm-0.out‚30844.58349609375$ memory_percent‚95.49925327301025* proc.memory.percent‚17.341136932373047 cpu‚26.102008655882138 disk.dm-1.in‚0" disk.dm-1.out‚4555.58349609375 disk.nvme0c0n1.in‚ 4.3359375' disk.nvme0c0n1.out‚27449.1552734375 disk.nvme1c1n1.in‚0 disk.md0.in‚0% disk./.usageGB‚1173.1315879821777 disk.rbd0.in‚0 disk.nvme0n1.in‚ 4.3359375& disk.nvme1c1n1.out‚8124.3818359375 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚8124.3818359375( proc.memory.availableMB‚ 737.40234375! proc.memory.rssMB‚ 2841.171875 network.recv‚812539) disk./.usagePercent‚76.41306549305831 disk.dm-0.in‚ 4.3359375 proc.cpu.threads‚17—@ŸöžÃ‚(:“ •ÌôÒÄòÐÌ gpu.0.gpu‚57.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚240.813' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚80.27099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%@/$2Ä- _wandbruntime‚2982  _runtime‚29829š‘þÉÅà trainer/global_step‚5799 % _timestamp‚1.7844895059442546e+09 _runtime‚2982.519236662 _step‚467 ' lr-AdamW/pg1‚0.00020444600786935114 ' lr-AdamW/pg2‚0.00020444600786935114êõwìÎÆÈ ' lr-AdamW/pg2‚0.00020444600786935114 trainer/global_step‚5799 % _timestamp‚1.7844895059442546e+09 _runtime‚2982.519236662 _step‚467 ' lr-AdamW/pg1‚0.00020444600786935114ÓQ7¶n2Ç- _wandbruntime‚2982  _runtime‚2982»ø¬é'ÈãO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30776232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚443961200790016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45203730 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 _step‚468 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚449530175737344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚449530534042112 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚443961183271424 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30776230 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103845621 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚444867974713336 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5568992465920 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5559318122536 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58641889 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45203730 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58631746 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15343646 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103835476 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5569333252096 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5568992465920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚297 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚439308656590800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 % _timestamp‚1.7844895059453773e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45203732 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5740849096704 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5569333252096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15343941 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5558978199214 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58641889 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46119876 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚358407817686528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚444867617750654 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚439308639551440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚352666835419648 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚352666810995200 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚449530534042112 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚443961200790016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45203730 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 _step‚468 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚449530175737344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚449530534042112 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚443961183271424 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30776230 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103845621 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚444867974713336 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5568992465920 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5559318122536 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58641889 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45203730 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58631746 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15343646 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103835476 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5569333252096 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5568992465920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚297 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚439308656590800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 % _timestamp‚1.7844895059453773e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45203732 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5740849096704 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784ÔÓÃú2Ê- _wandbruntime‚2982  _runtime‚2982­c×öò&ËìM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚444035392559104 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30781313 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45211232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚444942329019164 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚444941972056370 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚439382082673280 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45211234 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159684608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚449605655729664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚449605297417728 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46127426 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135260160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚358467491869184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.784489506323217e+09 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103863012 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚444035375040512 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5569922377216 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5570263170560 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚444035392559104 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58651778 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103863012 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5741939801600 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58641621 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 trainer/global_step‚5799 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46127720 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15346115 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 _runtime‚2982.898366241 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58641621 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0a¥›€÷&ÌñM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103863012 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚444035375040512 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5569922377216 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5570263170560 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚444035392559104 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58651778 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103863012 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5741939801600 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58641621 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 trainer/global_step‚5799 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46127720 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15346115 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 _runtime‚2982.898366241 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58641621 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚444035392559104 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30781313 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45211232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚444942329019164 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚444941972056370 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚439382082673280 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45211234 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159684608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚449605655729664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚449605297417728 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46127426 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135260160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚358467491869184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.784489506323217e+09 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935Õ( 2Í- _wandbruntime‚2982  _runtime‚2982Û ÏÎ 2loss/SRqqt1/assignment_loss‚0.5558520268392542 .loss/FRt2/detection_loss‚0.505504520060685 1loss/SRqqt1/detection_loss‚0.7331751369828364 1loss/SRqqt2/detection_loss‚0.8363024817479431 $ _timestamp‚1.784489506324285e+09 _runtime‚2982.898999632 /loss/FRt1/assignment_loss‚1.000956442801405 0loss/FRt2/assignment_loss‚2.1252726364282912 2loss/SRqqt2/assignment_loss‚0.5852943145291647 0loss/FBt1/detection_loss‚0.48068852087154107 /loss/FBt2/detection_loss‚0.7558238357305527 %loss/total_loss‚9.042816025709158 2loss/FBt1/assignment_loss‚0.002004373677986096 trainer/global_step‚5799 _step‚470 epoch‚7 2loss/FBt2/assignment_loss‚0.023159974320265925 /loss/FRt1/detection_loss‚1.4387817617192329Ô’>· φ epoch‚7 2loss/FBt2/assignment_loss‚0.023159974320265925 /loss/FRt1/detection_loss‚1.4387817617192329 2loss/SRqqt1/assignment_loss‚0.5558520268392542 .loss/FRt2/detection_loss‚0.505504520060685 1loss/SRqqt1/detection_loss‚0.7331751369828364 1loss/SRqqt2/detection_loss‚0.8363024817479431 $ _timestamp‚1.784489506324285e+09 _runtime‚2982.898999632 /loss/FRt1/assignment_loss‚1.000956442801405 0loss/FRt2/assignment_loss‚2.1252726364282912 2loss/SRqqt2/assignment_loss‚0.5852943145291647 0loss/FBt1/detection_loss‚0.48068852087154107 /loss/FBt2/detection_loss‚0.7558238357305527 %loss/total_loss‚9.042816025709158 2loss/FBt1/assignment_loss‚0.002004373677986096 trainer/global_step‚5799 _step‚470Öãp~_Ђ(:Ô ¤ÌôÒ­ƒéË disk.nvme1n1.in‚0 network.sent‚ 7.827656e+06$ disk./.usageGB‚1174.881217956543 disk.nvme0c0n1.in‚ 4.3359375( disk.nvme0c0n1.out‚27574.55224609375& disk.nvme0n1.out‚27574.55224609375 disk.md0.in‚0 disk.rbd0.out‚ 0.4697265625& disk.nvme1c1n1.out‚8125.1865234375# disk.dm-0.out‚30970.06884765625 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0 network.recv‚816750 disk.rbd0.in‚0 disk.nvme0n1.in‚ 4.3359375 disk.md0.out‚ 81.0390625 disk.dm-1.out‚ 4555.7421875( proc.memory.availableMB‚ 737.37109375) disk./.usagePercent‚76.52702934092447$ disk.nvme1n1.out‚8125.1865234375 disk.dm-0.in‚ 4.3359375$ memory_percent‚95.49944400787354! proc.memory.rssMB‚ 2841.171875* proc.memory.percent‚17.341136932373047 cpu‚26.100088896577663Dw&žÑ‚(:“ ¤ÌôÒ¡÷ÇÌ gpu.0.gpu‚90.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚235.389' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚78.46300000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù'þÔ¿Ò‚(:È ³ÌôÒÓÒË' disk.nvme0c0n1.out‚29942.8330078125* proc.memory.percent‚17.341136932373047 cpu‚26.098930344164096 disk.dm-0.in‚ 4.3359375 network.sent‚ 7.89185e+06' disk.nvme1c1n1.out‚8304.57275390625 disk.md0.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375! proc.memory.rssMB‚ 2841.171875% disk./.usageGB‚1175.7320137023926 disk.nvme1c1n1.in‚0% dis%è:\”k.nvme1n1.out‚8304.57275390625 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.md0.out‚81.4375 network.recv‚820799) disk./.usagePercent‚76.58244674824248 disk.nvme0n1.in‚ 4.3359375% disk.nvme0n1.out‚29942.8330078125 disk.nvme1n1.in‚0$ memory_percent‚95.50094604492188# proc.memory.availableMB‚737.125# disk.dm-0.out‚33516.00244140625 disk.dm-1.out‚4556.625ÄKùœÓ‚(:‘ ³ÌôÒ¾‹Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚236.095' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚78.69833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HÏ.ˆ2Ô- _wandbruntime‚3007  _runtime‚3007|uFÈÕ ' lr-AdamW/pg2‚0.00020427933939552056 trainer/global_step‚5849 $ _timestamp‚1.784489530735689e+09 _runtime‚3007.310755749 _step‚471 ' lr-AdamW/pg1‚0.000204279339395520566õ/ÍÖÇ ' lr-AdamW/pg1‚0.00020427933939552056 ' lr-AdamW/pg2‚0.00020427933939552056 trainer/global_step‚5849 $ _timestamp‚1.784489530735689e+09 _runtime‚3007.310755749 _step‚471×<¸·Â2×- _wandbruntime‚3007  _runtime‚3007ª'¥é'ØãO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45578832 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚453286292013568 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45578830 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59137491 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59127348 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚447670789244416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5788799262208 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104706178 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚453286292013568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚442979960714800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104716323 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5615520287744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59127348 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 % _timestamp‚1.7844895307378433e+09 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59137491 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15469760 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5788926140928 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚448585364939662 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚442979943675440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5615861073920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚361399578187776 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45578830 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5615520287744 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚447670771725824 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚355610627622400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 trainer/global_step‚5849 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31030280 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5615861073920×àîî'ÙèO @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚453286650318336 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46500329 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 % _timestamp‚1.7844895307378433e+09 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59137491 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15469760 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5788926140928 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚448585364939662 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚442979943675440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5615861073920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚361399578187776 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45578830 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5615520287744 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚447670771725824 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚355610627622400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 trainer/global_step‚5849 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31030280 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5615861073920 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45578832 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚453286292013568 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45578830 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59137491 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59127348 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚447670789244416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5788799262208 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104706178 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚453286292013568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚442979960714800 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104716323 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5615520287744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59127348 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59137223 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚453361413693952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5616450199040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5606349487458 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59137223 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5616790992384 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104723555 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45586334 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚443053386797280 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46507871 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45586334 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚447744981013504 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59147380 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5606689410892 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5616450199040 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚447744981013504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15472508 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚355669528379392 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 _step‚473 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104733714 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15472220 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚447744963494912 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46507581 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚447744963494912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚443053369757920 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 _runtime‚3007.670815929 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104733714 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5789758438400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935ÁË^ÙŽÜòM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59137223 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5616790992384 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104723555 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45586334 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚443053386797280 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46507871 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45586334 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚447744981013504 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59147380 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5606689410892 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5616450199040 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚447744981013504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15472508 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚355669528379392 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 _step‚473 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104733714 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15472220 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚447744963494912 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46507581 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚447744963494912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚443053369757920 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 _runtime‚3007.670815929 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104733714 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5789758438400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45586332 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚453361772005888 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59137223 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚453361413693952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5616450199040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5606349487458ÙE‘¦2Ý- _wandbruntime‚3007  _runtime‚3007÷9K¼Þþ 1loss/FBt1/assignment_loss‚0.01572398572844464 /loss/FBt1/detection_loss‚0.5476102600597577 .loss/FBt2/detection_loss‚0.521846755307067 % _timestamp‚1.7844895310966551e+09 _runtime‚ 3007.6714103 0loss/SRqqt1/detection_loss‚0.623666856574351 1loss/SRqqt2/detection_loss‚0.8567422563587693 0loss/FRt1/assignment_loss‚0.8996688002970681 0loss/FRt2/assignment_loss‚2.0309142289550732 3loss/FBt2/assignment_loss‚0.0013694288673683466 $loss/total_loss‚8.72016489875381 2loss/SRqqt1/assignment_loss‚0.6398379866920334 /loss/FRt1/detection_loss‚1.4197284654773243 /loss/FRt2/detection_loss‚0.5310081387383909 epoch‚7 trainer/global_step‚5849 _step‚474 2loss/SRqqt2/assignment_loss‚0.6320477356981624Ž-’ ߃ 0loss/FRt1/assignment_loss‚0.8996688002970681 0loss/FRt2/assignment_loss‚2.0309142289550732 3loss/FBt2/assignment_loss‚0.0013694288673683466 $loss/total_loss‚8.72016489875381 2loss/SRqqt1/assignment_loss‚0.6398379866920334 /loss/FRt1/detection_loss‚1.4197284654773243 /loss/FRt2/detection_loss‚0.5310081387383909 epoch‚7 trainer/global_step‚5849 _step‚474 2loss/SRqqt2/assignment_loss‚0.6320477356981624 1loss/FBt1/assignment_loss‚0.01572398572844464 /loss/FBt1/detection_loss‚0.5476102600597577 .loss/FBt2/detection_loss‚0.521846755307067 % _timestamp‚1.7844895310966551e+09 _runtime‚ 3007.6714103 0loss/SRqqt1/detection_loss‚0.623666856574351 1loss/SRqqt2/detection_loss‚0.8567422563587693Ú¯]tõ^à‚(:Ó ÂÌôÒÎßîË* proc.memory.percent‚17.341136932373047 proc.cpu.threads‚17 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375 network.sent‚ 7.907716e+06) disk./.usagePercent‚76.56576942104681$ disk./.usageGB‚1175.475975036621 disk.rbd0.in‚0 disk.nvme1n1.in‚0$ memory_percent‚95.49994468688965 network.recv‚824573 disk.md0.out‚81.78125 disk.dm-1.in‚0 disk.nvme0n1.in‚ 4.3359375! proc.memory.rssMB‚ 2841.171875 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375" disk.dm-0.out‚33735.3544921875 disk.dm-1.out‚ 4557.61328125 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚8353.81396484375% disk.nvme1n1.out‚8353.81396484375 cpu‚26.100582447183022( disk.nvme0c0n1.out‚30114.68017578125& disk.nvme0n1.out‚30114.68017578125' proc.memory.availableMB‚ 737.2890625¥ºÛmá‚(:’ ÂÌôÒÞç£Ì gpu.0.gpu‚82.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚145.81' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚48.60333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åú˜¹Nâ‚(:à ÑÌôÒ¦©ýË% disk./.usageGB‚1177.0398483276367 disk.dm-1.out‚ 4557.71875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375% disk.nvme0c0n1.out‚32149.78515625 disk.dm-0.in‚ 4.3359375" disk.dm-0.out‚35901.5341796875 disk.md0.in‚0 proc.memory.availableMB‚737# proc.memory.rssMB‚ 2841.26171875 proc.cpu.threads‚17# disk.nvme0n1.out‚32149.78515625 disk.nvme1n1.in‚0 disk.md0.out‚ 81.99609375" memory_percent‚95.501708984375) proc.memory.percent‚17.34168529510498 cpu‚26.101809503488145 network.sent‚ 7.959239e+06) disk./.usagePercent‚76.66763382691006 disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚8485.4775390625$ disk.nvme1n1.out‚8485.4775390625 network.recv‚828481Ë5Ëïžã‚(:“ ÑÌôÒõ³Ì gpu.0.gpu‚51.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚52.0 gpu.0.powerWatts‚193.303' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚64.43433333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>XLñ2ä- _wandbruntime‚3032  _runtime‚3032Nhƒ¸Éåà ' lr-AdamW/pg1‚0.00020411110357988955 ' lr-AdamW/pg2‚0.00020411110357988955 trainer/global_step‚5899 % _timestamp‚1.7844895560049736e+09 _runtime‚3032.580010205 _step‚475›ãÁhÎæÈ trainer/global_step‚5899 % _timestamp‚1.7844895560049736e+09 _runtime‚3032.580010205 _step‚475 ' lr-AdamW/pg1‚0.00020411110357988955 ' lr-AdamW/pg2‚0.00020411110357988955ÛJ/Õ»2ç- _wandbruntime‚3032  _runtime‚3032`fðé'èãO O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 % _timestamp‚1.7844895560060115e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚457042408289792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31284330 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5651864329230 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚457042408289792 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚446651264838800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5662048109568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚452303112128670 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5652204252552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59633093 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚364391206171136 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚457042766594560 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚452303469091352 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚358554468674048 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚451380360180224 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45953932 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚297 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59633093 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚457042766594560 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 105587025 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46880834 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45953930 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45953930 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15596799 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚364391363765760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚476 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45953932 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚451380377698816 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 _runtime‚3032.580910786 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚446651247799440 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105576880 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59622950 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚295 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31284332 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59622950 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5662048109568 trainer/global_step‚5899 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5836761921536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15596504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5662388895744 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5662388895744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚358554444249600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚457042408289792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31284330 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5651864329230 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚457042408289792 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚446651264838800 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5662048109568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚452303112128670 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5652204252552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59633093 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚364391206171136 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚457042766594560 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚452303469091352 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚358554468674048 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚451380360180224 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45953932 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚297 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59633093 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚457042766594560 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 105587025 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46880834 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45953930 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45953930 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15596799 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚364391363765760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚476 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45953932 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚451380377698816 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 _runtime‚3032.580910786 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚446651247799440 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105576880 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59622950 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚295 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31284332 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59622950 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5662048109568 trainer/global_step‚5899 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5836761921536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15596504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5662388895744 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5662388895744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 aMDeviceStatsMonitor.on_train_batch_start/inactùráqÄive_split_bytes.large_pool.freed‚358554444249600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚452377466434386 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚446724690921280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5662978020864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105604416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5663318814208 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59642982 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5837849533952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚358613320582144 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59642982 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚364451194540544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚446724673881920 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46888736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚364451039050240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31289413 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15599323 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45961434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 _runtime‚3032.956112711 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5652792552466 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 trainer/global_step‚5899 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46888451 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5653132475900 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 105604416 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15599040 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚451454569467904 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105594257 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59632825 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚03‘Ü÷&ìñM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 $ _timestamp‚1.784489556380941e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31289411 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5662978020864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚451454569467904 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45961434 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚451454551949312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105594257 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46888736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚364451039050240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31289413 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15599323 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45961434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 _runtime‚3032.956112711 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5652792552466 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 trainer/global_step‚5899 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46888451 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5653132475900 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 105604416 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15599040 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚451454569467904 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105594257 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59632825 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚452377466434386 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚446724690921280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5662978020864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105604416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5663318814208 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59642982 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5837849533952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚358613320582144 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59642982 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚364451194540544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚446724673881920 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460798524566016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60128695 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 trainer/global_step‚5949 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚301 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚450322568962800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31538382 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚455089966153216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚455089966153216 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 137364480 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚367382991469056 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 UA‚V¹ùi DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚460798524566016 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5708916717568 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚456020859317678 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚09-’Yî'÷èO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460798524566016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60128695 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 trainer/global_step‚5949 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚301 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚450322568962800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31538382 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚455089966153216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚455089966153216 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 137364480 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚367382991469056 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚460798524566016 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5708916717568 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚456020859317678 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106447582 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47261352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15722972 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5884730592256 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚456021216280360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚480 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5708575931392 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46336534 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5709505842688 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚456095570586188 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚450395995045280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46336532 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚367442984688128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3058.817949884 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚460873646246400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31543463 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5885823054336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7844895822427683e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚455164157922304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚305 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚361557137209344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60128427 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5885687794176 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106475118 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚455164140403712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5709505842688 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚361557161633792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60128427 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 _step‚481 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135260160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5699575540908 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚5949 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106464959 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚455164140403712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47269016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚450395978005920 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5699235617474 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15725555 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚460874004558336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46336532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚307 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159684608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46336534 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚367442825003520 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5709846636032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106464959 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434¿„ø&úòM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚456095213623394 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106475118 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5709846636032 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15725555 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚460874004558336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46336532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚307 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159684608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46336534 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚367442825003520 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5709846636032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106464959 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚455164157922304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60138584 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46336534 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5709505842688 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚456095570586188 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚450395995045280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46336532 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚367442984688128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3058.817949884 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚460873646246400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31543463 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5885823054336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7844895822427683e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚455164157922304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚305 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚361557137209344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60128427 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5885687794176 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106475118 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚455164140403712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5709505842688 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚361557161633792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60128427 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 _step‚481 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135260160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5699575540908 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚5949 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106464959 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚455164140403712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47269016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚450395978005920 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5699235617474 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46704132 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚453993856047440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5755103753216 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚464554640842240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5755444539392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31792430 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚453993873086800 trainer/global_step‚5999 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚464554999147008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31792432 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60614154 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5745090382568 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46704130 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚364442077504000 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 _step‚484 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚464554999147008 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5744750459246 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚458799537089024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107328429 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚458799554607616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15849963 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚370374939283456 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚364442101928448 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46704132 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚453993856047440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5755103753216 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚464554640842240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5755444539392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31792430 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚453993873086800 trainer/global_step‚5999 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚464554999147008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31792432 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60614154 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5745090382568 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46704130 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚364442077504000 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 _step‚484 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚464554999147008 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5744750459246 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240äAÀÓ2ˆ- _wandbruntime‚3084  _runtime‚3084ê5ßĉìM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15852542 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5756033664512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31797511 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107335661 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚464630120834560 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5756033664512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46711632 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚364500953836544 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚454067299169280 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107335661 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5933799006208 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚370434630165504 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122677248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚458873746376704 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5933676328960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚458873746376704 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5999 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚458873728858112 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107345820 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60624029 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46711634 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚459812960812402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5746018605916 $ _timestamp‚1.784489607793428e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47650053 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚364500978260992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 FC%°Ù. 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 _runtime‚3084.368624697 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107345820 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5756374457856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚370434777267200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60634186 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664‘‚÷&ŠñM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚370434630165504 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122677248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚458873746376704 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5933676328960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚458873746376704 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5999 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚458873728858112 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107345820 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60624029 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46711634 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚459812960812402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5746018605916 $ _timestamp‚1.784489607793428e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47650053 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚364500978260992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 _runtime‚3084.368624697 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107345820 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5756374457856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚370434777267200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60634186 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15852542 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5756033664512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31797511 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107335661 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚464630120834560 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5756033664512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46711632 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚364500953836544 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚454067299169280 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107335661 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5933799006208 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260å5·JB2‹- _wandbruntime‚3084  _runtime‚3084ƒÐU°Œ€ /loss/FBt2/detection_loss‚0.8045488607604057 %loss/total_loss‚9.494932729083281 epoch‚7 0loss/FRt1/assignment_loss‚0.8563857494757093 2loss/SRqqt1/assignment_loss‚0.5899110809168736 1loss/FBt2/assignment_loss‚0.05454234590124525 /loss/FRt2/detection_loss‚0.5247779572922487 trainer/global_step‚5999 % _timestamp‚1.7844896077945685e+09 2loss/SRqqt2/assignment_loss‚0.6314023143610878 0loss/SRqqt1/detection_loss‚0.622256243765212 /loss/FBt1/detection_loss‚0.6025889138965046 _runtime‚3084.369329648 _step‚486 /loss/FRt2/assignment_loss‚2.204963693549436 2loss/FBt1/assignment_loss‚0.008568510752856728 /loss/FRt1/detection_loss‚1.5037761238857237 1loss/SRqqt2/detection_loss‚1.0912109345259766?§¡ … 0loss/SRqqt1/detection_loss‚0.622256243765212 /loss/FBt1/detection_loss‚0.6025889138965046 _runtime‚3084.369329648 _step‚486 /loss/FRt2/assignment_loss‚2.204963693549436 2loss/FBt1/assignment_loss‚0.008568510752856728 /loss/FRt1/detection_loss‚1.5037761238857237 1loss/SRqqt2/detection_loss‚1.0912109345259766 /loss/FBt2/detection_loss‚0.8045488607604057 %loss/total_loss‚9.494932729083281 epoch‚7 0loss/FRt1/assignment_loss‚0.8563857494757093 2loss/SRqqt1/assignment_loss‚0.5899110809168736 1loss/FBt2/assignment_loss‚0.05454234590124525 /loss/FRt2/detection_loss‚0.5247779572922487 trainer/global_step‚5999 % _timestamp‚1.7844896077945685e+09 2loss/SRqqt2/assignment_loss‚0.6314023143610878æÏ$³ŸdŽ‚(:Ù ÍôÒœ±ÂÌ network.recv‚844094) disk./.usagePercent‚76.66800728334975 disk.md0.out‚ 83.54296875 disk.dm-0.in‚ 4.3359375 disk.nvme0c0n1.in‚ 4.3359375( disk.nvme0c0n1.out‚32236.80517578125 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0 disk.md0.in‚0! disk.dm-1.out‚4558.6787109375 disk.rbd0.in‚0' proc.memory.availableMB‚ 490.2734375) proc.memory.percent‚17.34168529510498 cpu‚26.100637390529307" disk.dm-0.out‚35989.1162109375 disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625' disk.nvme1c1n1.out‚8490.32080078125$ memory_percent‚97.00760841369629 proc.cpu.threads‚17 network.sent‚ 8.104405e+06$ disk./.usageGB‚1177.045581817627 disk.nvme1n1.in‚0% disk.nvme1n1.out‚8490.32080078125& disk.nvme0n1.out‚32236.80517578125# proc.memory.rssMB‚ 2841.26171875ʾöž‚(:“ ÍôÒŠ±ÕË gpu.0.gpu‚93.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚264.725' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.24166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ!„ý\‚(:Ñ œÍôÒàÕÐË disk.nvme0c0n1.in‚ 4.3359375' disk.nvme0c0n1.out‚32238.4775390625 network.sent‚ 8.15595e+06 network.recv‚848650% disk./.usageGB‚1177.0453186035156# disk.dm-0.out‚35990.46240234375 disk.dm-1.in‚0" disk.dm-1.out‚4558.92626953125 disk.md0.out‚ 83.96484375 disk.rbd0.out‚ 0.4697265625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 4.3359375% disk.nvme0n1.out‚32238.4775390625 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 8491.14453125$ memory_percent‚97.00632095336914 disk.dm-0.in‚ 4.3359375 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 8491.14453125 cpu‚26.10361571007056 proc.cpu.threads‚17) disk./.usagePercent‚76.66799013864292 disk.md0.in‚0& proc.memory.availableMB‚ 490.484375# proc.memory.rssMB‚ 2841.26171875) proc.memory.percent‚17.34168529510498B €ž‘‚(:“ œÍôÒ¶â‚Ì gpu.0.gpu‚58.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚240.248' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚80.08266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜÞkÇ2’- _wandbruntime‚3109  _runtime‚3109ïƞǓÁ % _timestamp‚1.7844896330345032e+09 _runtime‚3109.609495058 _step‚487 & lr-AdamW/pg1‚0.0002035970191538364 & lr-AdamW/pg2‚0.0002035970191538364 trainer/global_step‚6049¯Òˆ1Ì”Æ & lr-AdamW/pg1‚0.0002035970191538364 & lr-AdamW/pg2‚0.0002035970191538364 trainer/global_step‚6049 % _timestamp‚1.7844896330345032e+09 _runtime‚3109.609495058 _step‚487ç'Ý2•- _wandbruntime‚3109  _runtime‚31096~PËè'–âO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47079232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5791533447576 trainer/global_step‚6049 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32046480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚457665177210800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚462509143062016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚468311115423232 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 _step‚488 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61109756 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47079230 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚367385894131200 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5801631575040 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5791193524254 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5801972361216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚462509125543424 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15975955 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚457665160171440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚468310757118464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5980663373824 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5801631575040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚468311115423232 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 108188986 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61109756 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚373366719293952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 $ _timestamp‚1.784489633035547e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303[qkªT—çO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚468311115423232 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 108188986 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61109756 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚373366719293952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 $ _timestamp‚1.784489633035547e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47079232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5791533447576 trainer/global_step‚6049 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32046480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚457665177210800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚462509143062016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚468311115423232 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 _step‚488 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61109756 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47079230 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚367385894131200 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5801631575040 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5791193524254 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5801972361216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚462509125543424 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15975955 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚457665160171440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚468310757118464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5980663373824 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5801631575040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375è¥ëM£2˜- _wandbruntime‚3109  _runtime‚3109­æ™)ó&™íM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47086734 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚462583334831104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48029899 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47086732 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5981757011456 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7844896334077098e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108216522 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48029614 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5981623848448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚457738586253920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5802902279680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5802561486336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108216522 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 trainer/global_step‚6049 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108206363 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 108206363 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚468386237110784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5802902279680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32051563 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚462583334831104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133163008 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5792461670924 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚462583317312512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32051561 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚468385878798848 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47086734 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚367444770463744 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61129788 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚463530708001410 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5802561486336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15978053 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 _runtime‚3109.982831391 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61119631 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚462583317312512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚468386237110784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5792121747490 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚373426394312192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448îŒôø&šòM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108216522 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48029614 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5981623848448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚457738586253920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5802902279680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5802561486336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108216522 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 trainer/global_step‚6049 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108206363 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 108206363 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚468386237110784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5802902279680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32051563 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚462583334831104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133163008 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5792461670924 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚462583317312512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32051561 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚468385878798848 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47086734 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚367444770463744 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61129788 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚463530708001410 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5802561486336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15978053 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 _runtime‚3109.982831391 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61119631 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚462583317312512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚468386237110784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5792121747490 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚373426394312192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47086734 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚462583334831104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48029899 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47086732 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5981757011456 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7844896334077098e+09éÑœÔé2›- _wandbruntime‚3109  _runtime‚3109¯ÎW œƒ 1loss/SRqqt1/detection_loss‚0.5935675827356898 /loss/FBt2/detection_loss‚0.6047337456672851 %loss/total_loss‚9.155569854415862 2loss/SRqqt1/assignment_loss‚0.5545900626673478 2loss/SRqqt2/assignment_loss‚0.6691717998443447 /loss/FRt2/detection_loss‚0.6060408392735607 trainer/global_step‚6049 _runtime‚3109.983480302 0loss/FRt1/assignment_loss‚1.1567248413339257 0loss/FRt2/assignment_loss‚2.0334555313904743 /loss/FRt1/detection_loss‚1.4524537771321775 epoch‚7 $ _timestamp‚1.784489633408776e+09 _step‚490 3loss/FBt1/assignment_loss‚0.0013973763560196002 2loss/FBt2/assignment_loss‚0.012205467811857116 1loss/SRqqt2/detection_loss‚0.9693655050088933 /loss/FBt1/detection_loss‚0.5018633251942872IðS¶ˆ _runtime‚3109.983480302 0loss/FRt1/assignment_loss‚1.1567248413339257 0loss/FRt2/assignment_loss‚2.0334555313904743 /loss/FRt1/detection_loss‚1.4524537771321775 epoch‚7 $ _timestamp‚1.784489633408776e+09 _step‚490 3loss/FBt1/assignment_loss‚0.0013973763560196002 2loss/FBt2/assignment_loss‚0.012205467811857116 1loss/SRqqt2/detection_loss‚0.9693655050088933 /loss/FBt1/detection_loss‚0.5018633251942872 1loss/SRqqt1/detection_loss‚0.5935675827356898 /loss/FBt2/detection_loss‚0.6047337456672851 %loss/total_loss‚9.155569854415862 2loss/SRqqt1/assignment_loss‚0.5545900626673478 2loss/SRqqt2/assignment_loss‚0.6691717998443447 /loss/FRt2/detection_loss‚0.6060408392735607 trainer/global_step‚6049êóèyt_ž‚(:Ô «ÍôÒÅÁÒË disk.nvme0n1.in‚ 4.3359375 disk.nvme1n1.in‚0# disk.nvme1n1.out‚8491.833984375 disk.md0.in‚0 disk.md0.out‚ 84.3671875$ memory_percent‚97.00665473937988) proc.memory.percent‚17.34168529510498% disk.nvme1c1n1.out‚8491.833984375 network.recv‚851872( disk./.usagePercent‚76.5985466218348" disk.dm-1.out‚4559.07080078125 disk.rbd0.in‚0 disk.dm-0.in‚ 4.3359375 cpu‚26.106629508542344 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 4.3359375' disk.nvme0c0n1.out‚32267.9599609375# disk.dm-0.out‚36019.61865234375' proc.memory.availableMB‚ 490.4296875 disk.nvme1c1n1.in‚0 network.sent‚ 8.158872e+06% disk./.usageGB‚1175.9791870117188 disk.rbd0.out‚ 0.4697265625% disk.nvme0n1.out‚32267.9599609375# proc.memory.rssMB‚ 2841.26171875·ÃöžŸ‚(:“ «ÍôÒÄç±Ì gpu.0.gpu‚56.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚239.653' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚79.88433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MOÈè2 - _wandbruntime‚3134  _runtime‚3134‰¦¥øÆ¡À & lr-AdamW/pg1‚0.0002034225411677763 & lr-AdamW/pg2‚0.0002034225411677763 trainer/global_step‚6099 % _timestamp‚1.7844896583529167e+09 _runtime‚ 3134.92789114 _step‚491tވˢŠ& lr-AdamW/pg1‚0.0002034225411677763 & lr-AdamW/pg2‚0.0002034225411677763 trainer/global_step‚6099 % _timestamp‚1.7844896583529167e+09 _runtime‚ 3134.92789114 _step‚491ë98Q¢2£- _wandbruntime‚3134  _runtime‚3134|+‘å­¤âO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6028710529024 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47454330 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47454332 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61615501 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚472066873394688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6028579456000 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16101250 _step‚492 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109059688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5848159396864 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497472 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚466218713997824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16101538 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5837976512584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48401780 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚461336481334800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5848159396864 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47454330 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚3134.932450339 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61605358 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109069833 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚467174457847384 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚472067231699456 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚472066873394688 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61605358 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5837636589262 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48402070 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32300532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚370329710758400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0•<¿í'¥çO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47454330 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47454332 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61615501 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚472066873394688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6028579456000 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16101250 _step‚492 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109059688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5848159396864 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497472 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚466218713997824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16101538 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5837976512584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48401780 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚461336481334800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5848159396864 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47454330 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚3134.932450339 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61605358 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109069833 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚467174457847384 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚472067231699456 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚472066873394688 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61605358 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5837636589262 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48402070 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32300532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚370329710758400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6028710529024 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0ì‡ðpëb¦‚(:× ºÍôÒûÿžÌ network.recv‚856383 disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚32715.22900390625 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚8626.24267578125 disk.md0.in‚0 disk.md0.out‚ 84.75390625 disk.dm-0.in‚ 4.3359375 disk.nvme0c0n1.in‚ 4.3359375( disk.nvme0c0n1.out‚32715.22900390625' disk.nvme1c1n1.out‚8626.24267578125# proc.memory.rssMB‚ 2841.26171875) proc.memory.percent‚17.34168529510498 proc.cpu.threads‚17 network.sent‚ 8.222323e+06! disk.dm-1.out‚4559.6767578125 disk.rbd0.in‚0 disk.nvme1n1.in‚0" disk.dm-0.out‚36599.8662109375$ memory_percent‚97.00813293457031$ proc.memory.availableMB‚490.1875) disk./.usagePercent‚76.67250714770282% disk./.usageGB‚1177.1146659851074 disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625 cpu‚26.108890039327633¯™"Ÿ§‚(:” ºÍôÒ¡—ùÌ gpu.0.gpu‚91.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚52.0 gpu.0.powerWatts‚123.241' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚41.080333333333336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ)O[2¨- _wandbruntime‚3135  _runtime‚3135Îó±Àó&©íM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61615233 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6029536719360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚461409890377920 _runtime‚3135.299305545 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47461834 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109077065 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 % _timestamp‚1.7844896587240717e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109087224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚370388587090944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48409360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32305613 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6029667785216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5849430101504 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚466292923285504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5849089308160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚466292923285504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚376418123810304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚472142353387008 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚466292905766912 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109087224 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61625390 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚472141995075072 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48409645 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 _step‚493 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5849089308160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32305611 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47461832 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚370388611515392 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47461832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16103749 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚467248812153212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚461409907417280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61625390 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚376418279300608 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675›fýªòM WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚467248812153212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚461409907417280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61625390 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚376418279300608 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61615233 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6029536719360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚461409890377920 _runtime‚3135.299305545 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47461834 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109077065 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 % _timestamp‚1.7844896587240717e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109087224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚370388587090944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48409360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32305613 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6029667785216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 \JDeviceStatsMon&ñ^2witor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5849430101504 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚466292923285504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5849089308160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚466292923285504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚376418123810304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚472142353387008 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚466292905766912 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109087224 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61625390 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚472141995075072 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48409645 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 _step‚493 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5849089308160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32305611 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47461832 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚370388611515392 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47461832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16103749 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490í÷^Ö2«- _wandbruntime‚3135  _runtime‚3135äý…¬ÿ 2loss/FBt1/assignment_loss‚0.000886806040533499 0loss/FRt2/assignment_loss‚2.0448925841104733 0loss/SRqqt1/detection_loss‚0.675532760664451 1loss/SRqqt2/detection_loss‚0.9985757305050714 .loss/FBt1/detection_loss‚0.394146478057207 % _timestamp‚1.7844896587252042e+09 _step‚494 1loss/SRqqt1/assignment_loss‚0.606366765962616 2loss/SRqqt2/assignment_loss‚0.5387393299336086 .loss/FRt1/detection_loss‚1.288449253212822 /loss/FRt2/detection_loss‚0.5544378140409456 .loss/FBt2/detection_loss‚0.571311133038817 epoch‚7 0loss/FRt1/assignment_loss‚0.9385829447408875 3loss/FBt2/assignment_loss‚5.185108020208073e-05 %loss/total_loss‚8.611973451387634 trainer/global_step‚6099 _runtime‚3135.300105637Ÿ¢æ ­„ 0loss/FRt2/assignment_loss‚2.0448925841104733 0loss/SRqqt1/detection_loss‚0.675532760664451 1loss/SRqqt2/detection_loss‚0.9985757305050714 .loss/FBt1/detection_loss‚0.394146478057207 % _timestamp‚1.7844896587252042e+09 _step‚494 1loss/SRqqt1/assignment_loss‚0.606366765962616 2loss/SRqqt2/assignment_loss‚0.5387393299336086 .loss/FRt1/detection_loss‚1.288449253212822 /loss/FRt2/detection_loss‚0.5544378140409456 .loss/FBt2/detection_loss‚0.571311133038817 epoch‚7 0loss/FRt1/assignment_loss‚0.9385829447408875 3loss/FBt2/assignment_loss‚5.185108020208073e-05 %loss/total_loss‚8.611973451387634 trainer/global_step‚6099 _runtime‚3135.300105637 2loss/FBt1/assignment_loss‚0.000886806040533499îJï\®‚(:Ñ ÉÍôÒÓÀÌ disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1n1.in‚0 disk.md0.in‚0& proc.memory.availableMB‚ 490.359375 cpu‚26.11185438549176 network.sent‚ 8.285741e+06) disk./.usagePercent‚76.67279015960221% disk.nvme0c0n1.out‚33309.01953125# disk.nvme0n1.out‚33309.01953125' disk.nvme1c1n1.out‚8626.87646484375% disk.nvme1n1.out‚8626.87646484375 disk.dm-1.in‚0$ memory_percent‚97.00708389282227 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0 disk.md0.out‚ 85.0078125 disk.dm-0.in‚ 4.3359375! disk.dm-1.out‚4559.9580078125 proc.cpu.threads‚17 network.recv‚860439$ disk./.usageGB‚1177.119010925293# disk.dm-0.out‚37193.42919921875# proc.memory.rssMB‚ 2841.26171875) proc.memory.percent‚17.34168529510498’}C–ž¯‚(:“ ÉÍôÒïÙ´Ì gpu.0.gpu‚57.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚234.889' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚78.29633333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï£É2°- _wandbruntime‚3160  _runtime‚3160|öWÈɱà ' lr-AdamW/pg2‚0.00020324650956327853 trainer/global_step‚6149 % _timestamp‚1.7844896839469552e+09 _runtime‚3160.521948267 _step‚495 ' lr-AdamW/pg1‚0.00020324650956327853qÞ„Î²È trainer/global_step‚6149 % _timestamp‚1.7844896839469552e+09 _runtime‚3160.521948267 _step‚495 ' lr-AdamW/pg1‚0.00020324650956327853 ' lr-AdamW/pg2‚0.00020324650956327853ïd¸:ƒ2³- _wandbruntime‚3160  _runtime‚3160íùÝPé'´ãO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚470892205036392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5895028004864 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚373273527385600 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6076561837056 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5894687218688 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47829430 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚475823347975680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5895028004864 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159691776 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚379350089222656 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5894687218688 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5884419577592 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5884079654270 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47829432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32554582 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚469928302452224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 trainer/global_step‚6149 _runtime‚3160.525423454 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚475822989670912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48782247 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48781940 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16227665 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62100960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62111103 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62111103 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚465007785458800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109930390 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚475822989670912 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6076697104384 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚465007768419440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16227360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚469928319970816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47829430 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5895028004864 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159691776 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚379350089222656 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5894687218688 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5884419577592 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5884079654270 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47829432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32554582 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚469928302452224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 trainer/global_step‚6149 _runtime‚3160.525423454 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚475822989670912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48782247 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48781940 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16227665 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62100960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62111103 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62111103 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚465007785458800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109930390 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚475822989670912 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6076697104384 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚465007768419440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16227360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚469928319970816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47829430 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5895957923328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6077527366144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 _runtime‚3160.900405061 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109957926 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚465081211541280 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 % _timestamp‚1.7844896843252928e+09 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62110835 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47836932 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109957926 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109947767 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚470002494221312 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚379410082380288 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47836932 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5895617129984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocaä&ç· ted‚48789801 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5885347800940 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚470966202379426 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚470966559342220 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62120992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚465081194501920 _step‚497 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚475898111351296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚379409931084288 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109947767 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚475898469663232 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48789507 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16229846 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62120992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47836934 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚470002494221312 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚470002511739904 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚373332428142592 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6077654237696 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 trainer/global_step‚6149 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5885007877506 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62110835 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47836934)î¦>ø&¸òM EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62110835 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47836934 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚373332403718144 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5895957923328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6077527366144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 _runtime‚3160.900405061 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109957926 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚465081211541280 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 % _timestamp‚1.7844896843252928e+09 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62110835 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47836932 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109957926 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109947767 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚470002494221312 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚379410082380288 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47836932 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5895617129984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48789801 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5885347800940 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚470966202379426 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚470966559342220 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62120992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚465081194501920 _step‚497 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚475898111351296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚379409931084288 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109947767 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚475898469663232 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48789507 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16229846 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62120992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47836934 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚470002494221312 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚470002511739904 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚373332428142592 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6077654237696 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 trainer/global_step‚6149 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5885007877506 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ñç2¹- _wandbruntime‚3160  _runtime‚3160ÍÈðºÿ trainer/global_step‚6149 $ _timestamp‚1.784489684326374e+09 _step‚498 0loss/FRt2/assignment_loss‚1.8613125067729495 2loss/SRqqt2/assignment_loss‚0.5386842305864824 /loss/FRt1/detection_loss‚1.5041363461078294 /loss/FBt2/detection_loss‚0.6625572219729806 %loss/total_loss‚8.967233655035852 2loss/SRqqt1/assignment_loss‚0.6345896070025708 1loss/FBt1/assignment_loss‚0.01394859858351916 0loss/SRqqt2/detection_loss‚1.031083206224532 /loss/FBt1/detection_loss‚0.4904918023465394 epoch‚7 0loss/FRt1/assignment_loss‚0.8934628836089565 1loss/FBt2/assignment_loss‚0.08367761705662666 /loss/FRt2/detection_loss‚0.5936097116913251 _runtime‚3160.901506863 1loss/SRqqt1/detection_loss‚0.6596799230815409nc »„ epoch‚7 0loss/FRt1/assignment_loss‚0.8934628836089565 1loss/FBt2/assignment_loss‚0.08367761705662666 /loss/FRt2/detection_loss‚0.5936097116913251 _runtime‚3160.901506863 1loss/SRqqt1/detection_loss‚0.6596799230815409 trainer/global_step‚6149 $ _timestamp‚1.784489684326374e+09 _step‚498 0loss/FRt2/assignment_loss‚1.8613125067729495 2loss/SRqqt2/assignment_loss‚0.5386842305864824 /loss/FRt1/detection_loss‚1.5041363461078294 /loss/FBt2/detection_loss‚0.6625572219729806 %loss/total_loss‚8.967233655035852 2loss/SRqqt1/assignment_loss‚0.6345896070025708 1loss/FBt1/assignment_loss‚0.01394859858351916 0loss/SRqqt2/detection_loss‚1.031083206224532 /loss/FBt1/detection_loss‚0.4904918023465394òÓy]¼‚(:Ò ØÍôÒ¯«þË% disk./.usageGB‚1177.0743789672852# disk.nvme1n1.out‚8628.349609375 disk.md0.out‚ 85.421875 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚37199.81298828125 disk.rbd0.out‚ 0.4697265625% disk.nvme1c1n1.out‚8628.349609375 disk.rbd0.in‚0& disk.nvme0c0n1.out‚33314.986328125 disk.nvme0n1.in‚ 4.3359375( proc.memory.availableMB‚ 490.32421875# proc.memory.rssMB‚ 2841.26171875) proc.memory.percent‚17.34168529510498 proc.cpu.threads‚17" disk.dm-1.out‚4560.12939453125$ disk.nvme0n1.out‚33314.986328125 network.sent‚ 8.337865e+06 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 4.3359375$ memory_percent‚97.00729846954346 cpu‚26.11470865645559 network.recv‚864569) disk./.usagePercent‚76.66988301366456DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚473637890906624 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62606705 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124781568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6124589645824 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5930522719278 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚479579105947136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 trainer/global_step‚6199 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚474609595262718 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48204530 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48204532 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5941555826688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚468679089582800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 % _timestamp‚1.7844897083939679e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚473637890906624 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62596562 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 110801092 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62606705 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49162367 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376217368437248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚468679072543440 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 _step‚500 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚382341933658624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚479579464251904 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62596562 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48204530 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 _runtime‚3184.972388007 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48204532 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32808632 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚479579105947136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49162658 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32808632 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚479579105947136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49162658 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚473637890906624 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62606705 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124781568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6124589645824 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5930522719278 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚479579105947136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 trainer/global_step‚6199 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚474609595262718 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48204530 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48204532 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5941555826688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚468679089582800ô§=à72Æ- _wandbruntime‚3185  _runtime‚3185wi< ò&ÇìM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62616594 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6125675337216 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚479654227627520 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62606437 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚474683949568434 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 110828628 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48212032 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110828628 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚473712082675712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚382401920107008 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 trainer/global_step‚6199 $ _timestamp‚1.784489708752453e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 _runtime‚3185.327583506 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6125550562816 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5942485745152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48212032 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 110818469 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49169915 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚376276220345344 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚474684306531228 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚295 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5931450942514 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚501 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16356204 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5931790865948 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚468752498625920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5942144951808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32813711 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚468752515665280 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62606437 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚473712100194304 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚468752498625920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5942144951808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32813711 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚468752515665280 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62606437 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚473712100194304 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48212032 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110828628 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚473712082675712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚382401920107008 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 trainer/global_step‚6199 $ _timestamp‚1.784489708752453e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 _runtime‚3185.327583506 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6125550562816 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5942485745152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48212032 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 110818469 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49169915 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚376276220345344 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚474684306531228 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚295 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5931450942514 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚501 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16356204 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5931790865948 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49543291 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5987742862336 ZFDeviceStatsMonitor.on_train_batch_start/inacti–~£ #ve_split_bytes.all.freed‚385333807736832 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚472350376667440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚483335222223360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63092164 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63092164 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16480611 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚477347496879616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33062680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 _runtime‚3208.867476382 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚379161160640000 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16480901 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚472350393706800 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5977305707608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5987742862336 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48579632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _step‚504 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48579632 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚477347496879616 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122684416 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚483335222223360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49543583 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚478327699414408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5976965784286 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5988083648512 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49543291 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5987742862336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚385333807736832 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚472350376667440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚483335222223360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63092164 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63092164 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16480611 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚477347496879616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33062680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 _runtime‚3208.867476382 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚379161160640000 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16480901 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚472350393706800 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5977305707608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5987742862336 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48579632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _step‚504 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48579632 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚477347496879616 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122684416 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚483335222223360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49543583 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚478327699414408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5976965784286 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5988083648512 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 JæLÀò&ÕìM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 trainer/global_step‚6249 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111689171 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚385393796147200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚477421688648704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16483210 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6173603684352 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚472423819789280 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63112196 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚385393640656896 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚477421671130112 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 $ _timestamp‚1.784489732648754e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63112196 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111699330 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚379220061396992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚477421688648704 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48587132 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 _step‚505 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚483410702215680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6173734750208 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚472423802749920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33067763 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚379220036972544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚477421671130112 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5978233930956 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111689171 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚483410702215680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111699330 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49550971 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48587134 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚477421671130112 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5978233930956 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111689171 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚483410702215680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111699330 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49550971 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48587134 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111689171 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚385393796147200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚477421688648704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16483210 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6173603684352 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚472423819789280 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63112196 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚385393640656896 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚477421671130112 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 $ _timestamp‚1.784489732648754e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63112196 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111699330 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚379220061396992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚477421688648704 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48587132 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 _step‚505 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚483410702215680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6173734750208 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚472423802749920ùn/’,2×- _wandbruntime‚3209  _runtime‚3209ªW€Øû .loss/FRt2/detection_loss‚0.570970383960533 1loss/SRqqt1/detection_loss‚0.6059011424918539 1loss/SRqqt2/detection_loss‚0.8496654919304308 $ _timestamp‚1.784489732649693e+09 _step‚506 0loss/FRt2/assignment_loss‚2.0142894989827225 1loss/SRqqt1/assignment_loss‚0.436467645567863 /loss/FBt2/detection_loss‚0.5147124996527712 %loss/total_loss‚8.375318623857705 .loss/FBt2/assignment_loss‚0.00470554975696 /loss/FBt1/detection_loss‚0.5639642031642441 epoch‚7 0loss/FRt1/assignment_loss‚0.8156543290082532 trainer/global_step‚6249 _runtime‚3209.224401127 1loss/SRqqt2/assignment_loss‚0.636023659834426 2loss/FBt1/assignment_loss‚0.023668698759612593 /loss/FRt1/detection_loss‚1.3392955207480342ÜešôÙ€ 0loss/FRt1/assignment_loss‚0.8156543290082532 trainer/global_step‚6249 _runtime‚3209.224401127 1loss/SRqqt2/assignment_loss‚0.636023659834426 2loss/FBt1/assignment_loss‚0.023668698759612593 /loss/FRt1/detection_loss‚1.3392955207480342 .loss/FRt2/detection_loss‚0.570970383960533 1loss/SRqqt1/detection_loss‚0.6059011424918539 1loss/SRqqt2/detection_loss‚0.8496654919304308 $ _timestamp‚1.784489732649693e+09 _step‚506 0loss/FRt2/assignment_loss‚2.0142894989827225 1loss/SRqqt1/assignment_loss‚0.436467645567863 /loss/FBt2/detection_loss‚0.5147124996527712 %loss/total_loss‚8.375318623857705 .loss/FBt2/assignment_loss‚0.00470554975696 /loss/FBt1/detection_loss‚0.5639642031642441 epoch‚7ú2}Õ^Ú‚(:Ó …ÎôÒÝÝèË network.recv‚875675) disk./.usagePercent‚76.63890625556581 disk.rbd0.out‚ 0.4697265625 disk.md0.out‚ 86.65234375 disk.dm-0.in‚ 4.3359375$ memory_percent‚97.01089859008789) proc.memory.percent‚17.34168529510498 cpu‚26.10962083881402 network.sent‚ 8.45443e+06 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375( disk.nvme0c0n1.out‚33324.61474609375 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0# disk.nvme1n1.out‚8630.349609375# proc.memory.rssMB‚ 2841.26171875% disk./.usageGB‚1176.5988082885742 disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚33324.61474609375% disk.nvme1c1n1.out‚8630.349609375 disk.dm-1.in‚0 disk.md0.in‚0! disk.dm-0.out‚37208.490234375" disk.dm-1.out‚4560.45166015625& proc.memory.availableMB‚ 489.734375 proc.cpu.threads‚17ÚyAžÛ‚(:“ …ÎôÒ‡ØœÌ gpu.0.gpu‚70.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚251.398' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚83.79933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þPCçSÜ‚(:È ”ÎôÒû¡¿Ë# disk.nvme1n1.out‚8632.099609375 cpu‚26.107960952978047 disk.md0.in‚0 disk.dm-1.in‚0" disk.dm-1.out‚4560.62841796875 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.out‚ 33338.125 disk.nvme0n1.in‚ 4.3359375 disk.nvme0n1.out‚ 33338.125 disk.nvme1c1n1.in‚0$ memory_percent‚97.00722694396973) proc.memory.percent‚17.34168529510498 network.recv‚879842 disk.dm-0.in‚ 4.3359375% disk.nvme1c1n1.out‚8632.099609375 disk.nvme1n1.in‚0' proc.memory.availableMB‚ 490.3359375# proc.memory.rssMB‚ 2841.26171875 proc.cpu.threads‚17 network.sent‚ 8.515937e+06) disk./.usagePercent‚76.63894849614782% disk./.usageGB‚1176.5994567871094 disk.md0.out‚ 86.82421875# disk.dm-0.out‚37223.16162109375 disk.nvme0c0n1.in‚ 4.3359375} —Ý‚(:’ ”ÎôÒ滉Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚51.0 gpu.0.powerWatts‚117.461' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.153666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³þ¾÷2Þ- _wandbruntime‚3232  _runtime‚3232¦|ÄÈß trainer/global_step‚6299 % _timestamp‚1.7844897558017337e+09 _runtime‚ 3232.37668117 _step‚507 ' lr-AdamW/pg1‚0.00020270912135834395 ' lr-AdamW/pg2‚0.00020270912135834395ªêˆýÍàÇ ' lr-AdamW/pg2‚0.00020270912135834395 trainer/global_step‚6299 % _timestamp‚1.7844897558017337e+09 _runtime‚ 3232.37668117 _step‚507 ' lr-AdamW/pg1‚0.00020270912135834395û0‚>2á- _wandbruntime‚3232  _runtime‚3232¿/’yé'âãO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚388325819579904 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6220817888256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚482045446603416 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 112552641 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6034270684160 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚476021697830800 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6023748772616 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚481057067815424 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16606776 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6023408849294 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33316732 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7844897558026104e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6220697300992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚487091696804352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33316730 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚487091338499584 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16606476 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚382104977267200 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63597909 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6034270684160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6034611470336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚481057085334016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 145011712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚481057067815424 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48954732 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6220697300992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚487091696804352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33316730 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚487091338499584 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16606476 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚382104977267200 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63597909 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6034270684160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6034611470336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚481057085334016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 145011712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚481057067815424 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48954732 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16606776 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6023408849294 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33316732 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7844897558026104e+09ü ©)á2ä- _wandbruntime‚3232  _runtime‚3232¦üá&åìM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49931096 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚487166818491904 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6035200595456 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48962232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112570032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33321811 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.currenPaß÷Ì"t‚ 358311936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6035541388800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚476095106873920 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16609002 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6024337072530 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚476095123913280 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚482119800909244 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚487166818491904 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6221658791424 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚481131277103104 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚481131259584512 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 112559873 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63597641 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6221783565824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚482119443946450 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16609283 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49930813 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚388385661590016 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚382163878024192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6035541388800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33321813 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6024676995964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚481131277103104 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6035200595456 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚388385512391168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 112570032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚382163853599744 trainer/global_step‚6299 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48962234 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48962232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚481131259584512 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48962234 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚101577,)÷&æñM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 112570032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚382163853599744 trainer/global_step‚6299 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48962234 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48962232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚481131259584512 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48962234 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49931096 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚487166818491904 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6035200595456 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48962232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112570032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33321811 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6035541388800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚476095106873920 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16609002 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6024337072530 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚476095123913280 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚482119800909244 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚487166818491904 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6221658791424 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚481131277103104 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚481131259584512 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 112559873 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63597641 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6221783565824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚482119443946450 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16609283 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49930813 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚388385661590016 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚382163878024192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6035541388800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33321813 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6024676995964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚481131277103104 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6035200595456 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚388385512391168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720ýxÞ°«2ç- _wandbruntime‚3232  _runtime‚3232Ò挖 è„ 2loss/FBt2/assignment_loss‚0.014657390602863732 1loss/SRqqt1/detection_loss‚0.6699161197108423 /loss/FBt2/detection_loss‚0.4489353825471231 _runtime‚3232.737025896 .loss/FRt1/detection_loss‚1.344139664619521 1loss/SRqqt2/detection_loss‚1.0622098030109934 _step‚510 0loss/FRt1/assignment_loss‚0.8967446164576868 2loss/SRqqt1/assignment_loss‚0.5797234374304815 0loss/FBt1/detection_loss‚0.47928271202603356 %loss/total_loss‚8.720406165616001 trainer/global_step‚6299 % _timestamp‚1.7844897561623461e+09 2loss/SRqqt2/assignment_loss‚0.6567329469513095 3loss/FBt1/assignment_loss‚0.0001284726547719117 /loss/FRt2/detection_loss‚0.5606983906998465 epoch‚7 0loss/FRt2/assignment_loss‚2.0072372289045277Ø$GÑé‰ .loss/FRt1/detection_loss‚1.344139664619521 1loss/SRqqt2/detection_loss‚1.0622098030109934 _step‚510 0loss/FRt1/assignment_loss‚0.8967446164576868 2loss/SRqqt1/assignment_loss‚0.5797234374304815 0loss/FBt1/detection_loss‚0.47928271202603356 %loss/total_loss‚8.720406165616001 trainer/global_step‚6299 % _timestamp‚1.7844897561623461e+09 2loss/SRqqt2/assignment_loss‚0.6567329469513095 3loss/FBt1/assignment_loss‚0.0001284726547719117 /loss/FRt2/detection_loss‚0.5606983906998465 epoch‚7 0loss/FRt2/assignment_loss‚2.0072372289045277 2loss/FBt2/assignment_loss‚0.014657390602863732 1loss/SRqqt1/detection_loss‚0.6699161197108423 /loss/FBt2/detection_loss‚0.4489353825471231 _runtime‚3232.737025896þô6‹ÿžê‚(:“ £ÎôÒçÞÙÍ gpu.0.gpu‚71.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚239.479' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚79.82633333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï0‘eë‚(:Ú £ÎôÒ⎜Ò& disk.nvme0n1.out‚33347.24755859375' disk.nvme1c1n1.out‚8639.14794921875% disk.nvme1n1.out‚8639.14794921875 network.sent‚ 8.534697e+06 network.recv‚882961# disk.dm-0.out‚37238.10693359375& proc.memory.availableMB‚ 490.046875 cpu‚26.106120899538936$ disk./.usageGB‚1176.677661895752 disk.md0.in‚0 disk.dm-1.in‚0" disk.dm-1.out‚4560.74951171875 disk.rbd0.out‚ 0.4697265625 disk.nvme0n1.in‚ 4.3359375) proc.memory.percent‚17.34168529510498) disk./.usagePercent‚76.64404246186301 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0$ memory_percent‚97.00899124145508# proc.memory.rssMB‚ 2841.26171875 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 4.3359375( disk.nvme0c0n1.out‚33347.24755859375 disk.md0.out‚ 87.33984375 disk.dm-0.in‚ 4.3359375 disk.rbd0.in‚0¶¥|Gdì‚(:Ù ²ÎôÒÖÉÌ disk.dm-0.in‚ 4.3359375 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375 proc.cpu.threads‚17! disk.dm-1.out‚4560.8525390625 disk.rbd0.in‚0# proc.memory.rssMB‚ 2841.26171875) proc.memory.percent‚17.34168529510498 cpu‚26.104667044613336 network.recv‚887274% disk./.usageGB‚1176.7244606018066 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚8640.62939453125 disk.md0.out‚ 87.5703125 disk.dm-1.in‚0 disk.nvme0n1.in‚ 4.3359375% disk.nvme0n1.out‚33386.2763671875 network.sent‚ 8.584979e+06) disk./.usagePercent‚76.64709074103935 disk.nvme1n1.in‚0% disk.nvme1n1.out‚8640.62939453125# disk.dm-0.out‚37278.01513671875' disk.nvme0c0n1.out‚33386.2763671875$ memory_percent‚97.00901508331299( proc.memory.availableMB‚ 490.04296875 disk.md0.in‚0¼’q¬í‚(:’ ²ÎôÒ©ûË gpu.0.gpu‚70.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚251.39' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚83.79666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uèl2î- _wandbruntime‚3256  _runtime‚3256Vì<ÇïÁ _runtime‚3256.216403807 _step‚511 & lr-AdamW/pg1‚0.0002025269037232936 & lr-AdamW/pg2‚0.0002025269037232936 trainer/global_step‚6349 % _timestamp‚1.7844897796414468e+09ƒõ>”ÌðÆ % _timestamp‚1.7844897796414468e+09 _runtime‚3256.216403807 _step‚511 & lr-AdamW/pg1‚0.0002025269037232936 & lr-AdamW/pg2‚0.0002025269037232936 trainer/global_step‚6349ÿíaÚO2ñ- _wandbruntime‚3256  _runtime‚3256)EΡ•#òãO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49329832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6070191837624 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚479692984915440 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚490847813080576 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6081139292160 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚391317465620480 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6081139292160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49329832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚485762836829742 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16733948 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚490847454775808 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6080798505984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49329830 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚490847454775808 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49329830 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚479693001954800 _runtime‚3256.220403783 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚484766656269824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16733663 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64093511 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚385048818318848 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6268671726080 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33570782 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6268800701952 _step‚512 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 11341319 ·1ÕT8 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚484766673788416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚490847454775808 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6080798505984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49329830 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚490847454775808 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49329830 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚479693001954800 _runtime‚3256.220403783 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚484766656269824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16733663 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64093511 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚385048818318848 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6268671726080 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33570782 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6268800701952 _step‚512 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 113413198 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚484766673788416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚479692984915440 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚490847813080576 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6081139292160 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚391317465620480 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6081139292160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49329832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚485762836829742 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16733948 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49337334 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚490922934768128 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64103400 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16736218 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50312079 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚295 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚485837191135458 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6081728417280 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚484840848038912 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33575863 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚385107694651392 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113440734 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50312374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚385107670226944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚484840848038912 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚479766428037280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64093243 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49337334 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 137357312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113440734 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚484840865557504 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚490922934768128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6269627596800 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 % _timestamp‚1.7844897799992976e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚391377459605504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6269764954112®s„7ø&öòM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113430575 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 _runtime‚3256.574421406 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64103400 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49337334 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚490922934768128 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64103400 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16736218 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50312079 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚295 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚485837191135458 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6081728417280 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚484840848038912 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33575863 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚385107694651392 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113440734 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50312374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚385107670226944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚484840848038912 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚479766428037280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64093243 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49337334 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 137357312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113440734 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚484840865557504 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚490922934768128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6269627596800 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 % _timestamp‚1.7844897799992976e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚391377459605504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6269764954112 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1èÚ2÷- _wandbruntime‚3256  _runtime‚3256Ä =A øƒ 0loss/FRt2/assignment_loss‚2.0837238837132412 1loss/FBt2/assignment_loss‚0.07314059425001015 1loss/SRqqt2/detection_loss‚0.9808413588971359 0loss/FBt1/detection_loss‚0.49406683399401446 %loss/total_loss‚8.840103860941845 % _timestamp‚1.7844897800002537e+09 0loss/FRt1/assignment_loss‚0.9500025293844587 2loss/SRqqt2/assignment_loss‚0.6072606764271719 3loss/FBt1/assignment_loss‚0.0019494616543440505 /loss/FRt2/detection_loss‚0.4973287699201853 trainer/global_step‚6349 epoch‚7 2loss/SRqqt1/assignment_loss‚0.5823279292706355 .loss/FRt1/detection_loss‚1.355440385208246 1loss/SRqqt1/detection_loss‚0.6400085746562361 _runtime‚3256.578395232 _step‚514 /loss/FBt2/detection_loss‚0.5740128635661677ë .ùˆ 3loss/FBt1/assignment_loss‚0.0019494616543440505 /loss/FRt2/detection_loss‚0.4973287699201853 trainer/global_step‚6349 epoch‚7 2loss/SRqqt1/assignment_loss‚0.5823279292706355 .loss/FRt1/detection_loss‚1.355440385208246 1loss/SRqqt1/detection_loss‚0.6400085746562361 _runtime‚3256.578395232 _step‚514 /loss/FBt2/detection_loss‚0.5740128635661677 0loss/FRt2/assignment_loss‚2.0837238837132412 1loss/FBt2/assignment_loss‚0.07314059425001015 1loss/SRqqt2/detection_loss‚0.9808413588971359 0loss/FB5>skàt1/detection_loss‚0.49406683399401446 %loss/total_loss‚8.840103860941845 % _timestamp‚1.7844897800002537e+09 0loss/FRt1/assignment_loss‚0.9500025293844587 2loss/SRqqt2/assignment_loss‚0.6072606764271719‚\t §[ú‚(:Ð ÁÎôÒÍÀáË disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚8641.29443359375 proc.cpu.threads‚17 disk.nvme1n1.in‚0# disk.dm-0.out‚37300.23388671875 disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375! disk.nvme0n1.out‚ 33408.796875' proc.memory.availableMB‚ 490.2265625) proc.memory.percent‚17.34168529510498 network.sent‚ 8.588002e+06 network.recv‚890664% disk./.usageGB‚1176.9640464782715# disk.nvme0c0n1.out‚ 33408.796875$ memory_percent‚97.00789451599121# proc.memory.rssMB‚ 2841.26171875 cpu‚26.10305118692115% disk.nvme1n1.out‚8641.29443359375 disk.md0.out‚ 87.91796875! disk.dm-1.out‚4561.0361328125) disk./.usagePercent‚76.662696399822272e!›œû‚(:‘ ÁÎôÒìˆÆÌ gpu.0.gpu‚6.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚251.767' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚83.92233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘]©A2ü- _wandbruntime‚3280  _runtime‚3280{±×$Èý _runtime‚3280.115468122 _step‚515 ' lr-AdamW/pg1‚0.00020234314681525717 ' lr-AdamW/pg2‚0.00020234314681525717 trainer/global_step‚6399 $ _timestamp‚1.784489803540502e+09ž,nßÍþÇ trainer/global_step‚6399 $ _timestamp‚1.784489803540502e+09 _runtime‚3280.115468122 _step‚515 ' lr-AdamW/pg1‚0.00020234314681525717 ' lr-AdamW/pg2‚0.00020234314681525717ƒå*0 2ÿ- _wandbruntime‚3280  _runtime‚3280ô¾ã¢è'€âO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚387992610521600 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚483364306078800 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 trainer/global_step‚6399 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64589113 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16859921 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 $ _timestamp‚1.784489803541343e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚494603929356800 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 114294045 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49704930 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6316648230912 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6127667113984 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49704930 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64589113 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚494603571052032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚489480940981432 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6127326327808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64578970 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114283900 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49704932 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6116294979310 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚489480584018750 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚494603929356800 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50684751 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33824832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16860207 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚387992634946048 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50685039 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6127326327808 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49704932 _runtime‚3280.116191614 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚488476262242816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚488476244724224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6127667113984 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64578970 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚488476244724224,–Dí'çO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6116634902632 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49704930 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64589113 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚494603571052032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚489480940981432 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6127326327808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64578970 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114283900 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49704932 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6116294979310 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚489480584018750 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚494603929356800 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50684751 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33824832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16860207 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚387992634946048 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50685039 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6127326327808 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49704932 _runtime‚3280.116191614 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚488476262242816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚488476244724224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6127667113984 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64578970 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚488476244724224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚387992610521600 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚483364306078800 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 trainer/global_step‚6399 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64589113 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16859921 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 $ _timestamp‚1.784489803541343e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚494603929356800 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 114294045 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49704930 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6316648230912 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6127667113984 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚483437715121920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚494678692732416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚6399 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6128597032448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚517 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64599002 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6128597032448 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49712432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚494679051044352 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114311436 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6117563125980 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚489555295287260 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 $ _timestamp‚1.784489803899386e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚488550436493312 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49712432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49712434 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6128256239104 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64588845 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚488550436493312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114301277 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33829913 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49712434 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50692274 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33829911 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚394369094842368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚388051486854144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6128256239104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚483437732161280 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114301277 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚494679051044352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚05ʃ%ã„ñM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚488550436493312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114301277 VIDeviceStatsMonitor.on_train_batch_end/inactive_spli‹¶&t.large_pool.allocated‚33829913 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49712434 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50692274 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33829911 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚394369094842368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚388051486854144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6128256239104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚483437732161280 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114301277 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚494679051044352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 114311436 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16862363 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚483437715121920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚494678692732416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚6399 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6128597032448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚517 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64599002 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6128597032448 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49712432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚494679051044352 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114311436 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6117563125980 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚489555295287260 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 $ _timestamp‚1.784489803899386e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚488550436493312 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49712432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49712434 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6128256239104 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64588845 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360…y€Þ¢2…- _wandbruntime‚3280  _runtime‚3280piB5 †… 2loss/FBt1/assignment_loss‚0.003371662592544548 4loss/FBt2/assignment_loss‚0.00017120359546133295 0loss/SRqqt2/detection_loss‚1.031206915627273 /loss/FRt2/detection_loss‚0.5464576162970931 %loss/total_loss‚9.093858694389986 trainer/global_step‚6399 epoch‚7 2loss/SRqqt2/assignment_loss‚0.7538886258647528 1loss/SRqqt1/detection_loss‚0.6552409846808781 0loss/FBt1/detection_loss‚0.35465006405849836 % _timestamp‚1.7844898039003377e+09 _step‚518 0loss/FRt1/assignment_loss‚1.0398160078428769 /loss/FRt1/detection_loss‚1.3999221342653585 /loss/FBt2/detection_loss‚0.7928291019052267 _runtime‚3280.475022105 0loss/FRt2/assignment_loss‚1.9878066476221126 2loss/SRqqt1/assignment_loss‚0.5284977300379102B¦‡Š /loss/FRt1/detection_loss‚1.3999221342653585 /loss/FBt2/detection_loss‚0.7928291019052267 _runtime‚3280.475022105 0loss/FRt2/assignment_loss‚1.9878066476221126 2loss/SRqqt1/assignment_loss‚0.5284977300379102 2loss/FBt1/assignment_loss‚0.003371662592544548 4loss/FBt2/assignment_loss‚0.00017120359546133295 0loss/SRqqt2/detection_loss‚1.031206915627273 /loss/FRt2/detection_loss‚0.5464576162970931 %loss/total_loss‚9.093858694389986 trainer/global_step‚6399 epoch‚7 2loss/SRqqt2/assignment_loss‚0.7538886258647528 1loss/SRqqt1/detection_loss‚0.6552409846808781 0loss/FBt1/detection_loss‚0.35465006405849836 % _timestamp‚1.7844898039003377e+09 _step‚518 0loss/FRt1/assignment_loss‚1.0398160078428769†Üo³Zˆ‚(:Ï ÐÎôÒÍŠòË" disk.dm-1.out‚4564.35888671875 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375( disk.nvme0c0n1.out‚33583.80126953125 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 8669.92578125 disk.nvme1n1.in‚0 network.sent‚ 8.664618e+06 network.recv‚895394 disk.md0.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 4.3359375# proc.memory.rssMB‚ 2841.75390625 proc.cpu.threads‚17) disk./.usagePercent‚76.65857943392133 disk.dm-0.out‚ 37499.7578125" disk.nvme1n1.out‚ 8669.92578125$ proc.memory.availableMB‚489.6875) proc.memory.percent‚17.34468936920166% disk./.usageGB‚1176.9008407592773 disk.md0.out‚ 88.28515625 disk.dm-0.in‚ 4.3359375& disk.nvme0n1.out‚33583.80126953125$ memory_percent‚97.01118469238281 cpu‚26.102201546971642&A`”’‰‚(:‡ ÐÎôÒ ù¡Ì gpu.0.gpu‚39.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚160.992' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚53.664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07°Ñ–ZŠ‚(:Ï ßÎôÒÕšåË proc.cpu.threads‚17( disk.nvme0c0n1.out‚33584.77783203125& disk.nvme0n1.out‚33584.77783203125# disk.nvme1c1n1.out‚ 8725.9921875 cpu‚26.11305654937869 network.recv‚899198 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 88.68359375 disk.dm-0.out‚37555.80078125 disk.rbd0.out‚ 0.4697265625) proc.memory.percent‚17.34468936920166 network.sent‚ 8.715028e+06) disk./.usagePercent‚76.67825584243901 disk.nvme0c0n1.in‚ 4.3359375 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0$ memory_percent‚97.01266288757324' proc.memory.availableMB‚ 489.4453125$ disk./.usageGB‚1177.202922821045 disk.nvme0n1.in‚ 4.3359375! disk.nvme1n1.out‚ 8725.9921875" disk.dm-1.out‚4564.51513671875 disk.rbd0.in‚0# proc.memory.rssMB‚ 2841.75390625¯Ávž‹‚(:“ ßÎôÒŸØË gpu.0.gpu‚92.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚52.0 gpu.0.powerWatts‚117.137' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.04566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]X`S2Œ- _wandbruntime‚3302  _runtime‚3302†·=±{ =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9943604469299316 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.48030298948287964 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7561779618263245 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8909974098205566 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8377192616462708 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11742052435874939 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.753061056137085 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6515616178512573 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 particle/accuracy_4_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38264015316963196 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6866047978401184 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.42653918266296387 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1464988738298416 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.22499999403953552 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9925377368927002 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 6validation_average_jet_accuracy‚0.3552733063697815 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8464911580085754 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7350395917892456 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.645991861820221 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5475620031356812 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9804181456565857 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7520630955696106 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.025651423260569572 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 jet/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8596413731575012 particle/accuracy_2_of_2‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6847329139709473 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03695148974657059 *jet/accuracy_2_of_3‚0.5499298572540283 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03567197546362877 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.24463267624378204 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4442881941795349 .particle/accuracy_3_of_3‚0.999514102935791 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15120236575603485 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 *jet/accuracy_1_of_2‚0.7473225593566895 +jet/accuracy_3_of_4‚0.08482497185468674 % _timestamp‚1.7844898257724218e+09 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.654761016368866 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_5_of_5‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1701633185148239 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.710530161857605 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7699999809265137 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21893857419490814 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.49384912848472595 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9721560478210449 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9849520921707153 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +particle/sensitivity‚0.6433846354484558 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.887499988079071 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5266440510749817 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.659295380115509 particle/accuracy_1_of_3‚1 /particle/accuracy_4_of_4‚0.9555844068527222 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7534071207046509 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8508772850036621 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8742427825927734 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6688693761825562 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6515616178512573 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7285998463630676 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.14621689915657043 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1641159951686859 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17643648386001587 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8115037083625793 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6250072717666626 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9343185424804688 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4291283190250397 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5972543954849243 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 _runtime‚ 3302.34786673 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6145833730697632 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5971059203147888 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5808611512184143 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6478619575500488 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7039884924888611 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8442307710647583 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2715093195438385 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_5‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9860780835151672 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9962481260299683 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5164062976837158 particle/accuracy_2_of_3‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.22499999403953552 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5980954170227051 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1654956340789795 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9980574250221252 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6816580891609192 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6953064203262329 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4469697177410126 jet/accuracy_3_of_5‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9871968030929565 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03614635765552521 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.041986655443906784 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 particle/accuracy_1_of_5‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740737318992615 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03796318918466568 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.29940250515937805 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01616978645324707 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03207715228199959 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6349652409553528 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18938715755939484 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09352188557386398 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44999998807907104 +jet/accuracy_2_of_2‚0.15175768733024597 +particle/specificity‚0.9057062864303589 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978741407394409 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9953159093856812 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 _step‚519 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7347472906112671 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6938342452049255 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4553488790988922 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.13502642512321472 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6963603496551514 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9150067567825317 particle/accuracy_2_of_4‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7624674439430237 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 *jet/accuracy_1_of_1‚0.3134813904762268 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1014208197593689 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03041166625916958 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_3_of_4‚1 &particle/f_score‚0.647162914276123 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5282159447669983 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7632478475570679 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2615155279636383 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7699999809265137 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9961773753166199 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3840862810611725 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1895461231470108 )loss/val_total_loss‚8.476110871375193 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998833417892456 8"Purity/*FRt2SRqqt*FBt/event_purity‚0.215021014213562 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7096361517906189 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04841845855116844 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_3_of_3‚0.07141339033842087 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18959981203079224 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.7454230189323425 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14999307692050934 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6706538796424866 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6444582939147949 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1580774039030075 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8115037083625793 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9943354725837708 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7034802436828613 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3402777910232544 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7274633049964905 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8452991843223572 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6779430508613586 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16578607261180878 particle/accuracy_1_of_4‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3214285671710968 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17582352459430695 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03172312304377556 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3214285671710968 particle/accuracy_2_of_5‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5836357474327087 jet/accuracy_1_of_5‚1 trainer/global_step‚6407 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9783980846405029 *jet/accuracy_1_of_3‚0.9082897901535034 *jet/accuracy_1_of_4‚0.9535714387893677 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.16351798176765442 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13924849033355713 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8493768572807312 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17643648386001587 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10901448875665665 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13906249403953552 +jet/accuracy_4_of_4‚0.01666666567325592 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9889849424362183 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1890329271554947 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03207715228199959 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8392857313156128 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 epoch‚7 âÆm… Ž¶{ =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9943604469299316 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.48030298948287964 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7561779618263245 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8909974098205566 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8377192616462708 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11742052435874939 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.753061056137085 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6515616178512573 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 particle/accuracy_4_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38264015316963196 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6866047978401184 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.42653918266296387 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1464988738298416 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.22499999403953552 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9925377368927002 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 6validation_average_jet_accuracy‚0.3552733063697815 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8464911580085754 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7350395917892456 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.645991861820221 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5475620031356812 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9804181456565857 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7520630955696106 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.025651423260569572 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 jet/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8596413731575012 particle/accuracy_2_of_2‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6847329139709473 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03695148974657059 *jet/accuracy_2_of_3‚0.5499298572540283 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03567197546362877 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.24463267624378204 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4442881941795349 .particle/accuracy_3_of_3‚0.999514102935791 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15120236575603485 =&Purity/1FRt1SRqq‰‹õ=72t0FBt/event_proportion‚0.1467633992433548 *jet/accuracy_1_of_2‚0.7473225593566895 +jet/accuracy_3_of_4‚0.08482497185468674 % _timestamp‚1.7844898257724218e+09 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.654761016368866 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_5_of_5‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1701633185148239 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.710530161857605 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7699999809265137 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21893857419490814 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.49384912848472595 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9721560478210449 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9849520921707153 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +particle/sensitivity‚0.6433846354484558 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.887499988079071 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5266440510749817 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.659295380115509 particle/accuracy_1_of_3‚1 /particle/accuracy_4_of_4‚0.9555844068527222 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7534071207046509 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8508772850036621 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8742427825927734 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6688693761825562 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6515616178512573 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7285998463630676 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.14621689915657043 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1641159951686859 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17643648386001587 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8115037083625793 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6250072717666626 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9343185424804688 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4291283190250397 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5972543954849243 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 _runtime‚ 3302.34786673 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6145833730697632 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5971059203147888 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5808611512184143 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6478619575500488 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7039884924888611 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8442307710647583 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2715093195438385 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_5‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9860780835151672 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9962481260299683 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5164062976837158 particle/accuracy_2_of_3‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.22499999403953552 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5980954170227051 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1654956340789795 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9980574250221252 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6816580891609192 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6953064203262329 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4469697177410126 jet/accuracy_3_of_5‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9871968030929565 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03614635765552521 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.041986655443906784 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 particle/accuracy_1_of_5‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740737318992615 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03796318918466568 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.29940250515937805 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01616978645324707 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03207715228199959 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6349652409553528 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18938715755939484 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09352188557386398 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44999998807907104 +jet/accuracy_2_of_2‚0.15175768733024597 +particle/specificity‚0.9057062864303589 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978741407394409 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9953159093856812 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 _step‚519 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7347472906112671 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6938342452049255 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4553488790988922 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.13502642512321472 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6963603496551514 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9150067567825317 particle/accuracy_2_of_4‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7624674439430237 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 *jet/accuracy_1_of_1‚0.3134813904762268 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1014208197593689 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03041166625916958 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_3_of_4‚1 &particle/f_score‚0.647162914276123 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5282159447669983 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7632478475570679 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2615155279636383 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7699999809265137 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9961773753166199 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3840862810611725 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1895461231470108 )loss/val_total_loss‚8.476110871375193 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998833417892456 8"Purity/*FRt2SRqqt*FBt/event_purity‚0.215021014213562 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7096361517906189 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04841845855116844 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_3_of_3‚0.07141339033842087 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18959981203079224 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.7454230189323425 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14999307692050934 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6706538796424866 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6444582939147949 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1580774039030075 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8115037083625793 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9943354725837708 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7034802436828613 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3402777910232544 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7274633049964905 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8452991843223572 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6779430508613586 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16578607261180878 particle/accuracy_1_of_4‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3214285671710968 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17582352459430695 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03172312304377556 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3214285671710968 particle/accuracy_2_of_5‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5836357474327087 jet/accuracy_1_of_5‚1 trainer/global_step‚6407 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9783980846405029 *jet/accuracy_1_of_3‚0.9082897901535034 *jet/accuracy_1_of_4‚0.9535714387893677 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.16351798176765442 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13924849033355713 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8493768572807312 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17643648386001587 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10901448875665665 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13906249403953552 +jet/accuracy_4_of_4‚0.01666666567325592 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9889849424362183 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1890329271554947 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03207715228199959 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8392857313156128 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 epoch‚7‡.sç”_‚(:Ô îÎôÒ‡ûêË network.sent‚ 8.730835e+06 disk.nvme1c1n1.in‚0 disk.md0.out‚ 88.9921875" disk.dm-1.out‚4810.07666015625( proc.memory.availableMB‚ 740.87109375) proc.memory.percent‚17.34156608581543 cpu‚26.11262147262787& disk.nvme1c1n1.out‚9003.8447265625 network.recv‚902658 disk.nvme0c0n1.in‚ 4.3359375$ memory_percent‚95.47808170318604 disk.nvme0n1.in‚ 4.3359375 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚9003.8447265625 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0" proc.memory.rssMB‚ 2841.2421875 proc.cpu.threads‚17( disk./.usagePercent‚76.6740760126131% disk./.usageGB‚1177.1387519836426 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625( disk.nvme0c0n1.out‚33752.90478515625& disk.nvme0n1.out‚33752.90478515625 disk.dm-0.out‚37755.55078125$X7ž‚(:“ îÎôÒ²¨šÌ gpu.0.gpu‚71.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚249.155' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚83.05166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0q2‘- _wandbruntime‚3323  _runtime‚3323ý)É’Ã trainer/global_step‚6449 % _timestamp‚1.7844898466106431e+09 _runtime‚3323.187425633 _step‚520 ' lr-AdamW/pg1‚0.00020215785357747453 ' lr-AdamW/pg2‚0.00020215785357747453ÂÚÑFÎ“È ' lr-AdamW/pg1‚0.00020215785357747453 ' lr-AdamW/pg2‚0.00020215785357747453 trainer/global_step‚6449 % _timestamp‚1.7844898466106431e+09 _runtime‚3323.187425633 _step‚520ˆª®2”- _wandbruntime‚3323  _runtime‚3323ç°}%é'•ãO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51350208 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚6449 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6198370204672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6389404785152 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚495139892375068 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚500331194036736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚398918869328896 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50321196 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚494132841350656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34265360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17084846 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚495140249337750 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚494132841350656 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6187567971110 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 115569481 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65248285 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚500331194036736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6198710990848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50321196 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34265362 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50321194 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65238142 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 _runtime‚3323.187484573 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚494132823832064 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51350208 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚6449 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6198370204672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6389404785152 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚495139892375068 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚500331194036736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚398918869328896 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50321196 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚494132841350656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34265360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17084846 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚495140249337750 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚494132841350656 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6187567971110 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 115569481 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65248285 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚500331194036736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6198710990848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50321196 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34265362 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50321194 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65238142 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 _runtime‚3323.187484573 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚494132823832064 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚500406674029056 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50328696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚398978712003584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6390362745856 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6199300115968 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65248017 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6199640909312 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚494207015601152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135260160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17087062 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51357503 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚392588189573120 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115586872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65258174 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚6449 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65248017 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚494207015601152 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51357781 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115586872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚3323.541698197 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6199300115968 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚392588213997568 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6188496194458 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚494207033119744 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115576713 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34270443 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚495214603643578 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚489026107449120 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚500406674029056 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚522 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 $ _timestamp‚1.784489846966586e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34270441 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚494207033119744 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6199640909312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17087338 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6188156271024 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50328696 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51357503 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚392588189573120 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115586872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65258174 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚6449 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65248017 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚494207015601152 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51357781 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115586872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚3323.541698197 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6199300115968 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚392588213997568 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6188496194458 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚494207033119744 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115576713 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34270443 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚495214603643578 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚489026107449120 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚500406674029056 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚522 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 $ _timestamp‚1.784489846966586e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34270441 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚494207033119744 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6199640909312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17087338 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6188156271024 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50328696 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚500406674029056 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50328696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚398978712003584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6390362745856 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6199300115968 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65248017 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6199640909312 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚494207015601152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135260160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17087062Šçš2š- _wandbruntime‚3323  _runtime‚3323£¯]›‚ _step‚523 0loss/FRt2/assignment_loss‚1.9918272565176458 2loss/SRqqt1/assignment_loss‚0.5123889891777884 1loss/SRqqt2/detection_loss‚1.0336926920738603 /loss/FBt2/detection_loss‚1.0613940050825477 1loss/SRqqt1/detection_loss‚0.7251517147429891 /loss/FBt1/detection_loss‚0.6188539858767763 epoch‚8 .loss/FRt1/assignment_loss‚1.03792855477432 3loss/FBt2/assignment_loss‚0.0010283830471985312 /loss/FRt1/detection_loss‚1.4599489275002813 /loss/FRt2/detection_loss‚0.5131171367624112 trainer/global_step‚6449 _runtime‚3323.542632989 2loss/SRqqt2/assignment_loss‚0.7063109774774473 2loss/FBt1/assignment_loss‚0.005908365920028587 %loss/total_loss‚9.667550988953295 % _timestamp‚1.7844898469675786e+09£'ß< œ‡ 1loss/SRqqt1/detection_loss‚0.7251517147429891 /loss/FBt1/detection_loss‚0.6188539858767763 epoch‚8 .loss/FRt1/assignment_loss‚1.03792855477432 3loss/FBt2/assignment_loss‚0.0010283830471985312 /loss/FRt1/detection_loss‚1.4599489275002813 /loss/FRt2/detection_loss‚0.5131171367624112 trainer/global_step‚6449 _runtime‚3323.542632989 2loss/SRqqt2/assignment_loss‚0.7063109774774473 2loss/FBt1/assignment_loss‚0.005908365920028587 %loss/total_loss‚9.667550988953295 % _timestamp‚1.7844898469675786e+09 _step‚523 0loss/FRt2/assignment_loss‚1.9918272565176458 2loss/SRqqt1/assignment_loss‚0.5123889891777884 1loss/SRqqt2/detection_loss‚1.0336926920738603 /loss/FBt2/detection_loss‚1.0613940050825477‹Jãa‚(:Ö ýÎôÒþëëË network.recv‚906756 disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625 disk.md0.out‚ 89.38671875 disk.dm-0.out‚37756.16796875) proc.memory.percent‚17.34156608581543) disk./.usagePercent‚76.67408769089164" disk.dm-1.out‚5440.79150390625 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375& disk.nvme1c1n1.out‚9635.0576171875( proc.memory.availableMB‚ 740.54296875 network.sent‚ 8.774967e+06$ disk./.usageGB‚1177.138931274414 disk.rbd0.in‚0( disk.nvme0c0n1.out‚33753.87158203125& disk.nvme0n1.out‚33753.87158203125 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚9635.0576171875 disk.md0.in‚0 disk.nvme1c1n1.in‚0 disk.dm-0.in‚ 4.3359375$ memory_percent‚95.48008441925049" proc.memory.rssMB‚ 2841.2421875 cpu‚26.111144279876115 proc.cpu.threads‚17‰ìS“ž‚(:ˆ ýÎôÒɬŸÌ gpu.0.gpu‚70.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚256.731' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚85.577 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(}ÁcŸ‚(:Ø ŒÏôÒ±ÉÀË cpu‚26.109462605734066' disk.nvme1c1n1.out‚9793.00439453125 network.recv‚911248 disk.nvme0c0n1.in‚ 4.3359375% disk.nvme1n1.out‚9793.00439453125& proc.memory.availableMB‚ 740.515625& disk.nvme0n1.out‚33764.91845703125 disk.nvme1c1n1.in‚0 disk.dm-0.in‚ 4.3359375$ memory_percent‚95.48025131225586" proc.memory.rssMB‚ 2841.2421875 proc.cpu.threads‚17 disk.nvme0n1.in‚ 4.3359375 network.sent‚ 8.837319e+06) disk./.usagePercent‚76.67409365426792 disk.dm-1.in‚0! disk.dm-1.out‚5590.3466796875 disk.rbd0.out‚ 0.4697265625( disk.nvme0c0n1.out‚33764.91845703125 disk.nvme1n1.in‚0% disk./.usageGB‚1177.1390228271484 disk.rbd0.in‚0 disk.md0.in‚0 disk.md0.out‚ 89.75390625" disk.dm-0.out‚37774.8232421875) proc.memory.percent‚17.34156608581543©O, ‚(:’ ŒÏôÒ€Ä¸Ë gpu.0.gpu‚34.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚261.417' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.13899999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K÷Wê2¡- _wandbruntime‚3347  _runtime‚3347E o&Æ¢À _runtime‚ 3347.13930388 _step‚524 ' lr-AdamW/pg1‚0.00020197102697779292 ' lr-AdamW/pg2‚0.00020197102697779292 trainer/global_step‚6499 # _timestamp‚1.78448987056429e+09 íÍˣŠ' lr-AdamW/pg1‚0.00020197102697779292 ' lr-AdamW/pg2‚0.00020197102697779292 trainer/global_step‚6499 # _timestamp‚1.78448987056429e+09 _runtime‚ 3347.13930388 _step‚524Œ×nü52¤- _wandbruntime‚3347  _runtime‚33473Å Œ¥ãO G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚497842429805056 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚294 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17211155 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚395473154292224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6245238812672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚504087310312960 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50696296 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚401910626290176 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50696294 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚492623968451280 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 % _timestamp‚1.7844898705652099e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65733744 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚492623985490640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6234011036118 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116430038 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17211449 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚525 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34519412 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚504087668617728 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚497842429805056 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚497842412286464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6437347216384 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚395473129867776 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50696296 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65743887 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚504087310312960 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6245238812672 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116440183 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚498857996526758 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65733744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6244898026496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚6499 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 % _timestamp‚1.7844898705652099e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65733744 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚492623985490640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6234011036118 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116430038 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17211449 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚525 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34519412 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚504087668617728 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚497842429805056 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚497842412286464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6437347216384 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚395473129867776 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50696296 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65743887 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚504087310312960 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6245238812672 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116440183 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚498857996526758 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65733744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6244898026496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚6499 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚497842429805056 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚294 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17211155 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚395473154292224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6245238812672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚504087310312960 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50696296£e2§- _wandbruntime‚3347  _runtime‚3347¡WŠÌó&¨íM BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 _runtime‚3347.497987969 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚395532006200320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚401970308328960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚498932350832586 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6245827937792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6245827937792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚401970463819264 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6438433194496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6246168731136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚497916621574144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚526 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51738174 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚492697411573120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116457574 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116447415 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65753776 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚492697394533760 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50703796 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116457574 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65753776 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚504162790305280 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚504162431993344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚504162790305280 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6246168731136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6234939259466 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65743619 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚497916621574144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17213971 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚498931993869792 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6234599336032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚497916604055552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51738464 trainer/global_step‚6499 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834hmá©òM E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 _runtime‚3347.497987969 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚395532006200320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚401970308328960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚498932350832586 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6245827937792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6245827937792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚401970463819264 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6438433194496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6246168731136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚497916621574144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚526 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51738174 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚492697411573120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116457574 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116447415 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65753776 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚492697394533760 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50703796 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116457574 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65753776 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚504162790305280 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚504162431993344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_ʳ]train_batch_end/active_bytes.all.allocated‚504162790305280 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6246168731136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6234939259466 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65743619 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚497916621574144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17213971 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚498931993869792 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6234599336032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚497916604055552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51738464 trainer/global_step‚6499 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784Ž!í)2ª- _wandbruntime‚3347  _runtime‚3347äNÿW«€ 2loss/SRqqt2/assignment_loss‚0.5761158105491482 2loss/FBt2/assignment_loss‚0.002304702439039719 .loss/FRt2/detection_loss‚0.541309445275737 1loss/SRqqt2/detection_loss‚1.0535098434336585 epoch‚8 _runtime‚ 3347.49859933 _step‚527 0loss/FRt1/assignment_loss‚0.8880867403505486 /loss/FBt1/detection_loss‚0.5032145937735384 %loss/total_loss‚8.502658754346403 trainer/global_step‚6499 0loss/FRt2/assignment_loss‚1.9408048588717193 2loss/SRqqt1/assignment_loss‚0.5364197961053477 3loss/FBt1/assignment_loss‚0.0006649244925240923 % _timestamp‚1.7844898709238446e+09 /loss/FRt1/detection_loss‚1.3882891636123478 1loss/SRqqt1/detection_loss‚0.6661997087536029 -loss/FBt2/detection_loss‚0.40573916668919 7ç ¬… 0loss/FRt1/assignment_loss‚0.8880867403505486 /loss/FBt1/detection_loss‚0.5032145937735384 %loss/total_loss‚8.502658754346403 trainer/global_step‚6499 0loss/FRt2/assignment_loss‚1.9408048588717193 2loss/SRqqt1/assignment_loss‚0.5364197961053477 3loss/FBt1/assignment_loss‚0.0006649244925240923 % _timestamp‚1.7844898709238446e+09 /loss/FRt1/detection_loss‚1.3882891636123478 1loss/SRqqt1/detection_loss‚0.6661997087536029 -loss/FBt2/detection_loss‚0.40573916668919 2loss/SRqqt2/assignment_loss‚0.5761158105491482 2loss/FBt2/assignment_loss‚0.002304702439039719 .loss/FRt2/detection_loss‚0.541309445275737 1loss/SRqqt2/detection_loss‚1.0535098434336585 epoch‚8 _runtime‚ 3347.49859933 _step‚527wtúøc­‚(:Ø ›ÏôÒÝúêË network.sent‚ 8.842161e+06 disk.rbd0.in‚0( disk.nvme0c0n1.out‚33765.95166015625$ memory_percent‚95.47927379608154) proc.memory.percent‚17.34156608581543 cpu‚26.108007266673273 proc.cpu.threads‚17 network.recv‚914624) disk./.usagePercent‚76.65865969102714% disk./.usageGB‚1176.9020729064941 disk.nvme0n1.in‚ 4.3359375 disk.dm-1.in‚0! disk.dm-1.out‚5590.4287109375( proc.memory.availableMB‚ 740.67578125" proc.memory.rssMB‚ 2841.2421875& disk.nvme0n1.out‚33765.95166015625 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 90.109375 disk.dm-0.in‚ 4.3359375" disk.dm-0.out‚37775.8505859375 disk.rbd0.out‚ 0.4697265625' disk.nvme1c1n1.out‚9793.83447265625% disk.nvme1n1.out‚9793.83447265625 disk.nvme0c0n1.in‚ 4.3359375á«ÍPž®‚(:“ ›ÏôÒÎñØÌ gpu.0.gpu‚91.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚158.275' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚52.75833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}ñã2¯- _wandbruntime‚3371  _runtime‚3371ž·€½É°Ã ' lr-AdamW/pg2‚0.00020178267000861962 trainer/global_step‚6549 % _timestamp‚1.7844898944960463e+09 _runtime‚3371.071012201 _step‚528 ' lr-AdamW/pg1‚0.00020178267000861962ŒåÕFÎ±È _runtime‚3371.071012201 _step‚528 ' lr-AdamW/pg1‚0.00020178267000861962 ' lr-AdamW/pg2‚0.00020178267000861962 trainer/global_step‚6549 % _timestamp‚1.7844898944960463e+09‚µ2²- _wandbruntime‚3371  _runtime‚3371°”ÖÚé'³ãO G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66239489 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 _step‚529 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34773460 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6291766634496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚404902377661440 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66239489 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17337902 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚501552000740864 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52111362 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚502575743715766 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚502575386753084 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚404902215872512 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 137364480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6280454101126 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52111654 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚507843784893952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚507843426589184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117310885 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161788928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚398416970919424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚501552000740864 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚496295272575280 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6291766634496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66229346 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚398416946494976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 % _timestamp‚1.7844898944969485e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51071394 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6291425848320 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17338192 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6485269377536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6280114177804 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 _runtime‚3371.071860152 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚6549 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚507843426589184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51071396 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚501552018259456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 Pé¿î'´èO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚507843784893952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚507843426589184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117310885 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161788928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚398416970919424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚501552000740864 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚496295272575280 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6291766634496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66229346 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚398416946494976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 % _timestamp‚1.7844898944969485e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51071394 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6291425848320 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17338192 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6485269377536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6280114177804 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 _runtime‚3371.071860152 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚6549 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚507843426589184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51071396 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚501552018259456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66239489 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 _step‚529 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34773460 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6291766634496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚404902377661440 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66239489 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17337902 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚501552000740864 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52111362 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚502575743715766 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚502575386753084 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚404902215872512 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 137364480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6280454101126 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52111654 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117328276 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6292355759616 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51078896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66239221 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34778541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚530 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117328276 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51078898 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚501626192509952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚501626210028544 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚502649741058800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117318117 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117318117 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚507918906581504 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51078898 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚404962212556800 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66239221 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135260160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚496368698657760 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚398475847251968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6281042401040 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚496368715697120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66249378 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52119214 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34778543 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚404962052872192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52118925 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17340384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚398475822827520 % _timestamp‚1.7844898948532748e+09 _runtime‚3371.428446723 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66249378 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚507918548269568 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592ƒsCûø&·òM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚496368715697120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66249378 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52119214 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34778543 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚404962052872192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52118925 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17340384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚398475822827520 % _timestamp‚1.7844898948532748e+09 _runtime‚3371.428446723 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66249378 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚507918548269568 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117328276 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6292355759616 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51078896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66239221 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34778541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚530 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117328276 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51078898 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚501626192509952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚501626210028544 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚502649741058800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117318117 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117318117 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚507918906581504 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51078898 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚404962212556800 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66239221 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135260160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚496368698657760 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚398475847251968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6281042401040 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375’û€$a2¸- _wandbruntime‚3371  _runtime‚3371§UÌ7¹‚ 1loss/FBt1/assignment_loss‚0.02078639269552454 1loss/SRqqt1/detection_loss‚0.6622459163068255 % _timestamp‚1.7844898948549502e+09 0loss/FRt2/assignment_loss‚2.0989397008618145 2loss/SRqqt2/assignment_loss‚0.5728151448637595 trainer/global_step‚6549 _step‚531 0loss/FRt1/assignment_loss‚0.7779409263815199 /loss/FRt1/detection_loss‚1.3742714997064986 /loss/FRt2/detection_loss‚0.5029117813364344 /loss/FBt1/detection_loss‚0.5930440891358368 %loss/total_loss‚8.667048380518452 epoch‚8 _runtime‚3371.429753245 3loss/SRqqt1/assignment_loss‚0.49020304453112623 1loss/FBt2/assignment_loss‚0.03593645526119409 1loss/SRqqt2/detection_loss‚0.8722160402685404 /loss/FBt2/detection_loss‚0.6657373891693765Tñm º‡ 3loss/SRqqt1/assignment_loss‚0.49020304453112623 1loss/FBt2/assignment_loss‚0.03593645526119409 1loss/SRqqt2/detection_loss‚0.8722160402685404 /loss/FBt2/detection_loss‚0.6657373891693765 1loss/FBt1/assignment_loss‚0.02078639269552454 1loss/SRqqt1/detection_loss‚0.6622459163068255 % _timestamp‚1.7844898948549502e+09 0loss/FRt2/assignment_loss‚2.0989397008618145 2loss/SRqqt2/assignment_loss‚0.5728151448637595 trainer/global_step‚6549 _step‚531 0loss/FRt1/assignment_loss‚0.7779409263815199 /loss/FRt1/detection_loss‚1.3742714997064986 /loss/FRt2/detection_loss‚0.5029117813364344 /loss/FBt1/detection_loss‚0.5930440891358368 %loss/total_loss‚8.667048380518452 epoch‚8 _runtime‚3371.429753245“aJQxV»‚(:Ë ªÏôÒ…È”Ì" disk.dm-1.out‚5590.62841796875 disk.rbd0.in‚0& disk.nvme0n1.out‚33778.48876953125) proc.memory.percent‚17.34156608581543 proc.cpu.threads‚17 disk.md0.out‚90.4375 disk.dm-0.out‚ 37789.9609375# disk.nvme1c1n1.out‚ 9796.3359375$ memory_percent‚95.47953605651855' proc.memory.availableMB‚ 740.6328125) disk./.usagePercent‚76.65865596391697 disk.nvme1n1.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375 cpu‚26.106417406249847 network.recv‚918968% disk./.usageGB‚1176.9020156860352! disk.nvme1n1.out‚ 9796.3359375 disk.dm-0.in‚ 4.3359375( disk.nvme0c0n1.out‚33778.48876953125 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0" proc.memory.rssMB‚ 2841.2421875 network.sent‚ 8.918439e+06 disk.md0.in‚0ô¯/o‘¼‚(:† ªÏôÒå£ÎÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚52.0 gpu.0.powerWatts‚117.963' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.321 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯x•Ÿ\½‚(:Ñ ¹ÏôÒ¨ÌôË) proc.memory.percent‚17.34156608581543 network.sent‚ 8.968836e+06 disk.dm-0.in‚ 4.3359375 disk.rbd0.out‚ 0.4697265625% disk.nvme0n1.out‚33779.5791015625 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚9796.80908203125# memory_percent‚96.9839334487915" proc.memory.rssMB‚ 2841.2421875 network.recv‚923157 disk.md0.in‚0 disk.md0.out‚90.6875' disk.nvme0c0n1.out‚33779.5791015625 disk.nvme0n1.in‚ 4.3359375) disk./.usagePercent‚76.64318771126254 disk.rbd0.in‚0' disk.nvme1c1n1.out‚9796.80908203125 disk.nvme1n1.in‚0( proc.memory.availableMB‚ 494.15234375 cpu‚26.10631969337892 proc.cpu.threads‚17% disk./.usageGB‚1176.6645393371582 disk.dm-0.out‚37790.88671875 disk.dm-1.in‚0" disk.dm-1.out‚5590.70263671875 disk.nvme0c0n1.in‚ 4.3359375 ÏHÏž¾‚(:“ ¹ÏôÒ¾·¨Ì gpu.0.gpu‚92.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚247.378' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚82.45933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®Dâm2¿- _wandbruntime‚3394  _runtime‚3394–ÃdÿÉÀà ' lr-AdamW/pg1‚0.00020159278568687375 ' lr-AdamW/pg2‚0.00020159278568687375 trainer/global_step‚6599 % _timestamp‚1.7844899183664868e+09 _runtime‚3394.941492382 _step‚532%ºÀ=ÎÁÈ ' lr-AdamW/pg1‚0.00020159278568687375 ' lr-AdamW/pg2‚0.00020159278568687375 trainer/global_step‚6599 % _timestamp‚1.7844899183664868e+09 _runtime‚3394.941492382 _step‚532”Ý©sn2Â- _wandbruntime‚3394  _runtime‚3394åù:é'ÃãO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6326557242812 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚401360787546624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚401360763122176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚299 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52491508 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚407893998689280 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚505261606713856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35027512 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚407894152089600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17464297 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51446494 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚511599542865408 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6533235567104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚533 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚505261589195264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6338294456320 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66735091 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 118181587 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6326897166134 trainer/global_step‚6599 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚511599901170176 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚505261589195264 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚511599901170176 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 118171442 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51446494 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52491809 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 _runtime‚3394.944913788 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17463998 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51446496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚506293490904774 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚499966593738640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6338294456320 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚511599542865408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚301 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚506293133942092 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66724948 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6337953670144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66724948 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6337953670144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6533364542976 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143‚j´>ßÄèO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66724948 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6337953670144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6533364542976 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6326557242812 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚401360787546624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚401360763122176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚299 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52491508 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚407893998689280 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚505261606713856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35027512 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚407894152089600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17464297 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51446494 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚511599542865408 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6533235567104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚533 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚505261589195264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6338294456320 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66735091 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 N@DeviceStatsMonitor.on_train_batch_start/allocation.allU”.allocated‚ 118181587 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6326897166134 trainer/global_step‚6599 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚511599901170176 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚505261589195264 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚511599901170176 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 118171442 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51446494 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52491809 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 _runtime‚3394.944913788 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17463998 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51446496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚506293490904774 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚499966593738640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6338294456320 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚511599542865408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚301 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚506293133942092 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66724948 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6337953670144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0•öcyf2Å- _wandbruntime‚3395  _runtime‚3395 `¸hó&ÆíM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66744980 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚505335780964352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52499027 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17466714 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚401419663879168 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6327825389482 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118198978 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6534198675456 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51453998 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66734823 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚407953993620480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17466436 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 _step‚534 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚511674664545792 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚407953838130176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6534329741312 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35032591 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6327485466048 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚511675022857728 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 118188819 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚505335798482944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51453996 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52499307 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚505335798482944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35032593 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6338883581440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 % _timestamp‚1.7844899187262015e+09 _runtime‚3395.301368392 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 trainer/global_step‚6599 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6338883581440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66734823 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚505335780964352 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6339224374784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 118188819 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66744980 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚506367488247808 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51453998 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51453996 FÁNïø&ÇòM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚401419639454720 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚500040019821120 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚511674664545792 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚407953838130176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6534329741312 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35032591 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6327485466048 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚511675022857728 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 118188819 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚505335798482944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51453996 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52499307 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚505335798482944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35032593 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6338883581440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 % _timestamp‚1.7844899187262015e+09 _runtime‚3395.301368392 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 trainer/global_step‚6599 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6338883581440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66734823 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚505335780964352 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6339224374784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 118188819 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66744980 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚506367488247808 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51453998 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51453996 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52499027 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17466714 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚401419663879168 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6327825389482 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118198978 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6534198675456 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51453998 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66734823 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚407953993620480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17466436 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 _step‚534 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚510011238093782 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52871335 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6384481491968 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 _step‚537 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 % _timestamp‚1.7844899427892714e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚404304604173824 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6581145968640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚503637880823280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚515355659141632 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17589502 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚508971195168256 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51821594 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17589773 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 119042144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35281560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 _runtime‚3419.364153778 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚515356017446400 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52871062 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67230693 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚508971177649664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67220550 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚510010881131100 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 119052289 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51821596 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚508971177649664 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67230693 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6384822278144 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚515355659141632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35281562 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚404304579749376 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚515356017446400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124781568oI]\î'ÒèO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚510011238093782 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52871335 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6384481491968 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 _step‚537 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 % _timestamp‚1.7844899427892714e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚404304604173824 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6581145968640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚503637880823280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚515355659141632 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17589502 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚508971195168256 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51821594 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17589773 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 119042144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35281560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 _runtime‚3419.364153778 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚515356017446400 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52871062 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67230693 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚508971177649664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67220550 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚510010881131100 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 119052289 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51821596 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚508971177649664 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67230693 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6384822278144 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚515355659141632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35281562 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚404304579749376 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚515356017446400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124781568 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51821596 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚508971195168256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚410885725718016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67220550 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6384822278144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0™‹É<2Ó- _wandbruntime‚3419  _runtime‚3419rįÀò&ÔìM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35286641 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚404363456081920 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51829098 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚515431139133952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚509045369418752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119069680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52878862 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7844899431532853e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6373928531056 trainer/global_step‚6649 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67240582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51829098 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122677248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6374268454490 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119059521 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6385411403264 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119059521 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51829096 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17592219 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6582110989824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52878590 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚509045386937344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35286643 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6582233667072 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚510085592399610 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67230425õ<~`)ÕñM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6373928531056 trainer/global_step‚6649 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67240582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51829098 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122677248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6374268454490 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119059521 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6385411403264 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119059521 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51829096 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17592219 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6582110989824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52878590 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚509045386937344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35286643 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6582233667072 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚510085592399610 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_p·4`ôÎool.current‚ 545259520 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67230425 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35286641 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚404363456081920 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51829098 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚515431139133952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚509045369418752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119069680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52878862 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7844899431532853e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344š„bã2Ö- _wandbruntime‚3419  _runtime‚3419Ùj¢×ÿ /loss/FRt1/detection_loss‚1.5087038045223322 .loss/FBt2/detection_loss‚0.997886500088498 $loss/total_loss‚9.42198516209372 % _timestamp‚1.7844899431543865e+09 /loss/FRt2/assignment_loss‚2.025612237060244 1loss/FBt1/assignment_loss‚0.07166867167372391 0loss/SRqqt1/detection_loss‚0.622878677417221 1loss/SRqqt2/detection_loss‚0.9319900376329491 /loss/FBt1/detection_loss‚0.5899238401466529 epoch‚8 _step‚539 _runtime‚3419.729091911 2loss/SRqqt1/assignment_loss‚0.5985662203389955 2loss/SRqqt2/assignment_loss‚0.4713629512973649 /loss/FRt2/detection_loss‚0.6014826104136752 trainer/global_step‚6649 0loss/FRt1/assignment_loss‚0.9996608796138917 3loss/FBt2/assignment_loss‚0.0022487318881709534ê˱ž Ø„ % _timestamp‚1.7844899431543865e+09 /loss/FRt2/assignment_loss‚2.025612237060244 1loss/FBt1/assignment_loss‚0.07166867167372391 0loss/SRqqt1/detection_loss‚0.622878677417221 1loss/SRqqt2/detection_loss‚0.9319900376329491 /loss/FBt1/detection_loss‚0.5899238401466529 epoch‚8 _step‚539 _runtime‚3419.729091911 2loss/SRqqt1/assignment_loss‚0.5985662203389955 2loss/SRqqt2/assignment_loss‚0.4713629512973649 /loss/FRt2/detection_loss‚0.6014826104136752 trainer/global_step‚6649 0loss/FRt1/assignment_loss‚0.9996608796138917 3loss/FBt2/assignment_loss‚0.0022487318881709534 /loss/FRt1/detection_loss‚1.5087038045223322 .loss/FBt2/detection_loss‚0.997886500088498 $loss/total_loss‚9.42198516209372›ûaq–cÙ‚(:Ø ×ÏôÒÀððË disk.nvme0c0n1.in‚ 4.3359375% disk.nvme1n1.out‚9856.17529296875$ memory_percent‚96.98565006256104) proc.memory.percent‚17.34156608581543 network.recv‚930771 disk.dm-0.in‚ 4.3359375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚33796.00146484375 disk.nvme0n1.in‚ 4.3359375 disk.md0.in‚0 network.sent‚ 9.035157e+06$ disk./.usageGB‚1176.965103149414& disk.nvme0n1.out‚33796.00146484375 disk.nvme1n1.in‚0 disk.md0.out‚ 91.59765625( proc.memory.availableMB‚ 493.87109375" proc.memory.rssMB‚ 2841.2421875 proc.cpu.threads‚17) disk./.usagePercent‚76.66276522712353 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚9856.17529296875 cpu‚26.10881309204613" disk.dm-0.out‚37864.3779296875 disk.dm-1.in‚0! disk.dm-1.out‚5591.0439453125 disk.rbd0.out‚ 0.46972656255ÚF “Ú‚(:ˆ ×ÏôÒ–·¢Ì gpu.0.gpu‚57.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚265.728' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚88.576 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g¶¼'cÛ‚(:Ø æÏôÒÔ¿þË disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625' disk.nvme1c1n1.out‚9856.86474609375 disk.nvme1n1.in‚0 network.sent‚ 9.098581e+06) disk./.usagePercent‚76.66273590719015 disk.md0.in‚0 disk.md0.out‚ 91.9609375# disk.dm-0.out‚37866.70361328125! disk.dm-1.out‚5591.2216796875' disk.nvme0c0n1.out‚33798.5966796875 disk.nvme1c1n1.in‚0 network.recv‚935289% disk./.usageGB‚1176.9646530151367% disk.nvme0n1.out‚33798.5966796875" proc.memory.rssMB‚ 2841.2421875 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375% disk.nvme1n1.out‚9856.86474609375$ memory_percent‚96.98460102081299( proc.memory.availableMB‚ 494.04296875) proc.memory.percent‚17.34156608581543 cpu‚26.107613258388533 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚04,ÍwžÜ‚(:“ æÏôÒ¡í±Ì gpu.0.gpu‚91.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚227.678' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚75.89266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®2ˆO2Ý- _wandbruntime‚3443  _runtime‚3443mO”ÈÞ ' lr-AdamW/pg1‚0.00020120844717561067 ' lr-AdamW/pg2‚0.00020120844717561067 trainer/global_step‚6699 % _timestamp‚1.7844899668717332e+09 _runtime‚ 3443.44665265 _step‚540¿1ÍßÇ ' lr-AdamW/pg1‚0.00020120844717561067 ' lr-AdamW/pg2‚0.00020120844717561067 trainer/global_step‚6699 % _timestamp‚1.7844899668717332e+09 _runtime‚ 3443.44665265 _step‚540œê¤uC2à- _wandbruntime‚3443  _runtime‚3443y…Jé'áãO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _runtime‚3443.447382462 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚512680766104064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17715217 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52196696 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6431009313792 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚519112133722624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52196696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚519112133722624 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6431350099968 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52196694 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67716152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35535612 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚519111775417856 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52196694 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚519111775417856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6431350099968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6629124880896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚407248420801024 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚507309201986640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚513728628320108 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53250829 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67716152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35535612 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚519111775417856 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52196694 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚519111775417856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6431350099968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6629124880896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚407248420801024 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚507309201986640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚513728628320108 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53250829 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _runtime‚3443.447382462 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚512680766104064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17715217 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52196696 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6431009313792žÓì 2ã- _wandbruntime‚3443  _runtime‚3443D¥8¸ˆ äíM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53258338 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6630216228352 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 % _timestamp‚1.7844899672291465e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚512754957873152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53258044 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17717645 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 =*ªckUHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6420711519498 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52204196 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚413937513361920 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6431939225088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚407307297133568 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚519187255410176 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52204198 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚512754975391744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119940382 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚519187255410176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35540693 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3443.804340338 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚519186897098240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6420371596064 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17717353 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚507382628069120 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52204198 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119940382 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119930223 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚407307272709120 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67726027 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6630085162496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67736184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6431939225088:A Vø&åòM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6431939225088 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53258338 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6630216228352 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 % _timestamp‚1.7844899672291465e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚512754957873152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53258044 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17717645 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6420711519498 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52204196 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚413937513361920 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6431939225088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚407307297133568 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚519187255410176 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52204198 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚512754975391744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119940382 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚519187255410176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35540693 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3443.804340338 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚519186897098240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6420371596064 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17717353 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚507382628069120 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52204198 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119940382 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119930223 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚407307272709120 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67726027 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6630085162496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67736184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157žJGÖ2æ- _wandbruntime‚3443  _runtime‚3443&:?çÿ /loss/FRt2/detection_loss‚0.5674917029890817 1loss/SRqqt1/detection_loss‚0.6576068252366511 1loss/SRqqt2/detection_loss‚0.9216187433369315 %loss/total_loss‚8.790562911621265 _step‚543 0loss/FRt2/assignment_loss‚1.8627232919613868 2loss/SRqqt1/assignment_loss‚0.6556917905400041 2loss/FBt2/assignment_loss‚0.005215549648511342 /loss/FBt1/detection_loss‚0.4226537255104631 epoch‚8 0loss/FRt1/assignment_loss‚0.8853910392893977 2loss/SRqqt2/assignment_loss‚0.6191364418526898 /loss/FBt1/assignment_loss‚0.063365638210477 trainer/global_step‚6699 _runtime‚ 3443.80557743 /loss/FBt2/detection_loss‚0.6977244253503159 % _timestamp‚1.7844899672308533e+09 /loss/FRt1/detection_loss‚1.43194373769535523s¿I è„ /loss/FBt2/detection_loss‚0.6977244253503159 % _timestamp‚1.7844899672308533e+09 /loss/FRt1/detection_loss‚1.4319437376953552 /loss/FRt2/detection_loss‚0.5674917029890817 1loss/SRqqt1/detection_loss‚0.6576068252366511 1loss/SRqqt2/detection_loss‚0.9216187433369315 %loss/total_loss‚8.790562911621265 _step‚543 0loss/FRt2/assignment_loss‚1.8627232919613868 2loss/SRqqt1/assignment_loss‚0.6556917905400041 2loss/FBt2/assignment_loss‚0.005215549648511342 /loss/FBt1/detection_loss‚0.4226537255104631 epoch‚8 0loss/FRt1/assignment_loss‚0.8853910392893977 2loss/SRqqt2/assignment_loss‚0.6191364418526898 /loss/FBt1/assignment_loss‚0.063365638210477 trainer/global_step‚6699 _runtime‚ 3443.80557743Ÿü‡—Ú^é‚(:Ó õÏôÒƒ±ÎË$ disk.nvme1n1.out‚9906.2998046875 cpu‚26.10592627994266 proc.cpu.threads‚17 network.sent‚ 9.101503e+06' disk./.usagePercent‚76.658551356358 disk.md0.in‚0$ memory_percent‚96.98648452758789! disk.dm-1.out‚5591.2998046875& disk.nvme0n1.out‚33944.39501953125 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚9906.2998046875 disk.md0.out‚ 92.3671875# disk.dm-0.out‚38060.98876953125" proc.memory.rssMB‚ 2841.2421875 network.recv‚938326% disk./.usageGB‚1176.9004096984863 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625( disk.nvme0c0n1.out‚33944.39501953125 disk.dm-0.in‚ 4.3359375& proc.memory.availableMB‚ 493.734375) proc.memory.percent‚17.34156608581543 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 disk.nvme1n1.in‚0M¸´žê‚(:“ õÏôÒŽ³±Ì gpu.0.gpu‚79.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚249.487' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚83.16233333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_ƒáÅIë‚(:¾ „ÐôÒÌœÈË' disk./.usagePercent‚76.675045558207$ disk./.usageGB‚1177.153636932373 disk.nvme0n1.in‚ 4.3359375! disk.dm-1.out‚5591.4580078125 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625) proc.memory.percent‚17.34156608581543 disk.md0.in‚0 disk.nvme0c0n1.out‚34035( proc.memory.availableMB‚ 493.78515625 proc.cpu.threads‚17# disk.dm-0.out‚38171.52587890625 network.sent‚ 9.165712e+06 network.recv‚942678 disk.nvme0n1.out‚34035' disk.nvme1c1n1.out‚9927.04150390625 disk.md0.out‚ 92.6640625 disk.dm-1.in‚0" proc.memory.rssMB‚ 2841.2421875 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0% disk.nvme1n1.out‚9927.04150390625 disk.dm-0.in‚ 4.3359375 disk.nvme0c0n1.in‚ 4.3359375$ memory_percent‚96.98617458343506 cpu‚26.10439237495243˜ÖJLœì‚(:‘ „ÐôÒ¾–€Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚52.0 gpu.0.powerWatts‚118.674' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.55800000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×K¢2í- _wandbruntime‚3467  _runtime‚3467XwÅxÉîà ' lr-AdamW/pg1‚0.00020101399914205479 ' lr-AdamW/pg2‚0.00020101399914205479 trainer/global_step‚6749 % _timestamp‚1.7844899908149054e+09 _runtime‚3467.389860659 _step‚544Ä,÷!ÎïÈ _runtime‚3467.389860659 _step‚544 ' lr-AdamW/pg1‚0.00020101399914205479 ' lr-AdamW/pg2‚0.00020101399914205479 trainer/global_step‚6749 % _timestamp‚1.7844899908149054e+09 (4¸D2ð- _wandbruntime‚3467  _runtime‚3467FZ¯è'ñâO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6477537135616 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52571794 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚510980489071280 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚522867891694080 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52571796 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6677221160960 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚295 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚416869458589184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚517446375509116 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120793693 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚ 3467.39096713 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53630756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17841096 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120783548 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚516390372077056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6677094282240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 % _timestamp‚1.7844899908158169e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35789660 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚410192237428224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚410192213003776 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6477537135616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚416869307286016 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68211754 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35789662 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚517446732471798 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚522867891694080 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52571794 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6466226361158 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚522868249998848 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚516390354558464 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚510980506110640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360ƒOÃ/òçO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35789660 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚410192237428224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚410192213003776 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6477537135616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚416869307286016 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68211754 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.¬SªN¾"large_pool.allocated‚35789662 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚517446732471798 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚522867891694080 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52571794 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6466226361158 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚522868249998848 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚516390354558464 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚510980506110640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6477537135616 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52571794 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚510980489071280 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚522867891694080 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52571796 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6677221160960 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚295 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚416869458589184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚517446375509116 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120793693 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚ 3467.39096713 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53630756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17841096 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120783548 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚516390372077056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6677094282240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 % _timestamp‚1.7844899908158169e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143¡\C!2ó- _wandbruntime‚3467  _runtime‚3467Ÿ.]ò&ôìM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚516464546327552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784489991174489e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚516464546327552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68231786 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17843889 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚416929296886272 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚522943013374464 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17843612 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚517521086777626 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6467154584506 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚511053932193120 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6678052059648 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 _step‚546 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚516464563846144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52579298 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120811084 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚410251089336320 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52579298 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6478807840256 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6478467046912 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6478807840256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚511053915153760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120800925 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚522943371686400 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53638632 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53638353 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68221629 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52579296 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68231786 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68221629 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35794741 trainer/global_step‚6749 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 120811084 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚3467.749636596 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6478467046912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35794743 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚410251113760768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120800925 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚416929141395968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520ŒH] ÷&õñM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52579296 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68231786 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68221629 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35794741 trainer/global_step‚6749 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 120811084 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚3467.749636596 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6478467046912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35794743 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚410251113760768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120800925 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚416929141395968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚516464546327552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784489991174489e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚516464546327552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68231786 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17843889 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚416929296886272 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚522943013374464 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17843612 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚517521086777626 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6467154584506 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚511053932193120 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6678052059648 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 _step‚546 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚516464563846144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52579298 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120811084 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚410251089336320 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52579298 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6478807840256 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6478467046912 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6478807840256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚511053915153760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120800925 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚522943371686400 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53638632 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53638353 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68221629¢ÀÚŠÑ2ö- _wandbruntime‚3467  _runtime‚3467àEc-÷‚ 0loss/FRt2/assignment_loss‚2.0745547579162884 1loss/SRqqt2/detection_loss‚1.0055833866228223 .loss/FBt2/detection_loss‚0.770285160387201 %loss/total_loss‚8.578551032578485 epoch‚8 $ _timestamp‚1.784489991175424e+09 2loss/SRqqt1/assignment_loss‚0.5174893111740008 /loss/FRt2/detection_loss‚0.5373758247630316 0loss/FRt1/assignment_loss‚0.8528607289669877 2loss/SRqqt2/assignment_loss‚0.5383337689815341 /loss/FRt1/detection_loss‚1.3065656663155405 0loss/SRqqt1/detection_loss‚0.628370858423534 0loss/FBt1/detection_loss‚0.33080572213593606 trainer/global_step‚6749 _runtime‚3467.750074467 2loss/FBt1/assignment_loss‚0.012793590637682945 3loss/FBt2/assignment_loss‚0.0035322562539257824 _step‚547Ì ø‡ 2loss/SRqqt1/assignment_loss‚0.5174893111740008 /loss/FRt2/detection_loss‚0.5373758247630316 0loss/FRt1/assignment_loss‚0.8528607289669877 2loss/SRqqt2/assignment_loss‚0.5383337689815341 /loss/FRt1/detection_loss‚1.3065656663155405 0loss/SRqqt1/detection_loss‚0.628370858423534 0loss/FBt1/detection_loss‚0.33080572213593606 trainer/global_step‚6749 _runtime‚3467.750074467 2loss/FBt1/assignment_loss‚0.012793590637682945 3loss/FBt2/assignment_loss‚0.0035322562539257824 _step‚547 0loss/FRt2/assignment_loss‚2.0745547579162884 1loss/SRqqt2/detection_loss‚1.0055833866228223 .loss/FBt2/detection_loss‚0.770285160387201 %loss/total_loss‚8.578551032578485 epoch‚8 $ _timestamp‚1.784489991175424e+09£<Ú=dù‚(:Ù “ÐôÒøÒóË( disk.nvme0c0n1.out‚34141.70166015625% disk.nvme1n1.out‚9965.27978515625 disk.md0.out‚ 93.08984375" proc.memory.rssMB‚ 2841.2421875) proc.memory.percent‚17.34156608581543) disk./.usagePercent‚76.67407477024304! disk.dm-1.out‚5591.6298828125 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 cpu‚26.10286558939876 proc.cpu.threads‚17 network.sent‚ 9.181578e+06 network.recv‚946288% disk./.usageGB‚1177.1387329101562' disk.nvme1c1n1.out‚9965.27978515625 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0& disk.nvme0n1.out‚34141.70166015625" disk.dm-0.out‚38315.3720703125$ memory_percent‚96.98655605316162( proc.memory.availableMB‚ 493.72265625 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375¼0{(“ú‚(:ˆ “ÐôÒ‘š¨Ì gpu.0.gpu‚82.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚265.623' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚88.541 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0C… 2û- _wandbruntime‚3491  _runtime‚3491-—’áÈü ' lr-AdamW/pg1‚0.00020081803606775047 ' lr-AdamW/pg2‚0.00020081803606775047 trainer/global_step‚6799 $ _timestamp‚1.784490014486785e+09 _runtime‚3491.061735453 _step‚548òK"“ÍýÇ ' lr-AdamW/pg1‚0.00020081803606775047 ' lr-AdamW/pg2‚0.00020081803606775047 trainer/global_step‚6799 $ _timestamp‚1.784490014486785e+09 _runtime‚3491.061735453 _step‚548¤ß Ö2þ- _wandbruntime‚3491  _runtime‚3491T@ŽÿãO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6725076166656 E?DeviceStatsMonitor.on_train_bat }V$[&ch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 _step‚549 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54011701 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚521164479660806 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6512669426166 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚526624366275072 % _timestamp‚1.7844900144876294e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚520099960531456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52946896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚419861105797632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6512329502844 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 140817408 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 116392960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6524064957440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36043710 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚419861246615040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚413136029630976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚526624366275072 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17967709 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52946894 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚520099943012864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68717499 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6524405743616 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚520099943012864 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚514651810234640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6725192559616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68707356 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6524064957440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6524405743616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17967989 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68717499 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54011419 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 trainer/global_step‚6799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚526624007970304 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52946894 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935:íàyî'€èO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 140817408 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 116392960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6524064957440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36043710 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚419861246615040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚413136029630976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚526624366275072 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17967709 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52946894 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚520099943012864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68717499 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6524405743616 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚520099943012864 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚514651810234640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6725192559616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68707356 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6524064957440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6524405743616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17967989 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68717499 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54011419 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 trainer/global_step‚6799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚526624007970304 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52946894 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6725076166656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 _step‚549 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54011701 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚521164479660806 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6512669426166 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚526624366275072 % _timestamp‚1.7844900144876294e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚520099960531456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52946896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚419861105797632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6512329502844¥EA-2- _wandbruntime‚3491  _runtime‚3491#-ûÉó&‚íM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6513597649514 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 % _timestamp‚1.7844900148450596e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17970516 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 trainer/global_step‚6799 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6726164457984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68727388 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17970234 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚413194905963520 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52954398 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 121671627 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 _runtime‚3491.420205731 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6525335662080 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68727388 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚419921094845952 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚514725219277760 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6524994868736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6726029197824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36048793 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚520174134781952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6524994868736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚521238833966634 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚413194930387968 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121681786 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52954396 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54019025 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚514725236317120 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚520174134781952 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54019309 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚526699487962624 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68717231 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52954396 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚520174152300544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6525335662080 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52954398 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚526699487962624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 trainer/global_step‚6799 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6726164457984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68727388 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17970234 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚413194905963520 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52954398 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 121671627 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 _runtime‚3491.420205731 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6525335662080 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68727388 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚419921094845952 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚514725219277760 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6524994868736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6726029197824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36048793 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requedí]“‰sted_bytes.all.peak‚ 12030241375 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚520174134781952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6524994868736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚521238833966634 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚413194930387968 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121681786 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52954396 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54019025 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚514725236317120 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚520174134781952 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54019309 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚526699487962624 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68717231 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52954396 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚520174152300544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6525335662080 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52954398 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚526699487962624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6513597649514 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 % _timestamp‚1.7844900148450596e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17970516 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0¦‹Üêò2„- _wandbruntime‚3491  _runtime‚34915knÅ…þ 2loss/SRqqt1/assignment_loss‚0.6039883090145312 0loss/FBt1/detection_loss‚0.49717358147044344 /loss/FBt2/detection_loss‚0.8524555712273079 trainer/global_step‚6799 _runtime‚3491.420725241 _step‚551 2loss/SRqqt2/assignment_loss‚0.5270362224554235 1loss/SRqqt1/detection_loss‚0.6127275370129225 /loss/SRqqt2/detection_loss‚1.00304547346465 %loss/total_loss‚8.960656601901473 /loss/FRt1/assignment_loss‚1.002358667384413 2loss/FBt2/assignment_loss‚0.008695616948445815 /loss/FRt2/detection_loss‚0.5284848955159906 epoch‚8 0loss/FRt2/assignment_loss‚1.8569972045572893 0loss/FBt1/assignment_loss‚0.0333510555259345 .loss/FRt1/detection_loss‚1.434342467324122 % _timestamp‚1.7844900148460095e+09”»K¹ †ƒ /loss/FBt2/detection_loss‚0.8524555712273079 trainer/global_step‚6799 _runtime‚3491.420725241 _step‚551 2loss/SRqqt2/assignment_loss‚0.5270362224554235 1loss/SRqqt1/detection_loss‚0.6127275370129225 /loss/SRqqt2/detection_loss‚1.00304547346465 %loss/total_loss‚8.960656601901473 /loss/FRt1/assignment_loss‚1.002358667384413 2loss/FBt2/assignment_loss‚0.008695616948445815 /loss/FRt2/detection_loss‚0.5284848955159906 epoch‚8 0loss/FRt2/assignment_loss‚1.8569972045572893 0loss/FBt1/assignment_loss‚0.0333510555259345 .loss/FRt1/detection_loss‚1.434342467324122 % _timestamp‚1.7844900148460095e+09 2loss/SRqqt1/assignment_loss‚0.6039883090145312 0loss/FBt1/detection_loss‚0.49717358147044344§õ\þù\‡‚(:Ñ ¢ÐôÒéúìË% disk./.usageGB‚1177.0944328308105 disk.nvme0n1.in‚ 4.3359375 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625( proc.memory.availableMB‚ 493.40234375" disk.nvme0n1.out‚ 34154.1171875' disk.nvme1c1n1.out‚9968.53173828125) proc.memory.percent‚17.34156608581543 cpu‚26.10150043799284 proc.cpu.threads‚17) disk./.usagePercent‚76.67118924154441 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0% disk.nvme1n1.out‚9968.53173828125 disk.md0.out‚ 93.43359375! disk.dm-0.out‚38330.193359375" disk.dm-1.out‚5591.73291015625 network.sent‚ 9.243841e+06 network.recv‚950597 disk.nvme0c0n1.in‚ 4.3359375$ disk.nvme0c0n1.out‚ 34154.1171875$ memory_percent‚96.98851108551025" proc.memory.rssMB‚ 2841.242187518מˆ‚(:“ ¢ÐôÒøƒ¥Ì gpu.0.gpu‚95.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚140.961' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚46.98700000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö'“‰‚(:ˆ ±ÐôÒÆ÷ëË gpu.0.gpu‚94.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚253.338' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚84.446 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´E^Š‚(:Ó ±ÐôÒŒœ©Ì network.recv‚953828 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375$ memory_percent‚96.98710441589355$ disk./.usageGB‚1177.094841003418 disk.md0.out‚ 93.984375! disk.dm-0.out‚38331.048828125" disk.dm-1.out‚5591.82275390625' proc.memory.availableMB‚ 493.6328125) proc.memory.percent‚17.34156608581543 cpu‚26.099855026071985 proc.cpu.threads‚17$ disk.nvme1n1.out‚9969.6142578125 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625& disk.nvme0n1.out‚34155.15283203125 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚9969.6142578125" proc.memory.rssMB‚ 2841.2421875 network.sent‚ 9.29428e+06) disk./.usagePercent‚76.67121582826367( disk.nvme0c0n1.out‚34155.15283203125§Ò&û2‹- _wandbruntime‚3515  _runtime‚3515!,¬nɌà ' lr-AdamW/pg1‚0.00020062056109144418 ' lr-AdamW/pg2‚0.00020062056109144418 trainer/global_step‚6849 % _timestamp‚1.7844900384578667e+09 _runtime‚3515.034451874 _step‚552:.•ÎÈ ' lr-AdamW/pg1‚0.00020062056109144418 ' lr-AdamW/pg2‚0.00020062056109144418 trainer/global_step‚6849 % _timestamp‚1.7844900384578667e+09 _runtime‚3515.034451874 _step‚552¨;K$2Ž- _wandbruntime‚3515  _runtime‚3515H]ûé'ãO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122524952 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53321994 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚523809531467264 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 trainer/global_step‚6849 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53321996 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚524882226849814 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69202958 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18094653 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚524881869887132 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53321994 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36297760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚416079870682624 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 % _timestamp‚1.7844900384584177e+09 _runtime‚3515.034509754 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚523809548985856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6773101962752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54392413 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69213101 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚523809548985856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6570592779264 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54392716 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚422852972645376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚553 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚518323097319280 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚530380482551296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36297762 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69213101 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚530380124246528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159691776 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚422852812953600 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚530380124246528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18094954 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚416079846258176 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53321996 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚301 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚530380482551296 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6570933565440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122535097 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6570592779264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6558772567852 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69202958 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 135267328 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6772966695424 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322‘Àß;î'èO JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69213101 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚530380124246528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159691776 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚422852812953600 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚530380124246528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18094954 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚416079846258176 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53321996 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚301 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚530380482551296 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6570933565440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122535097 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6570592779264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6558772567852 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69202958 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 135267328 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6772966695424 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122524952 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53321994 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚523809531467264 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 trainer/global_step‚6849 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53321996 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚524882226849814 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69202958 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18094653 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚524881869887132 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53321994 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36297760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚416079870682624 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 % _timestamp‚1.7844900384584177e+09 _runtime‚3515.034509754 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚523809548985856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6773101962752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54392413 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69213101 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚523809548985856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6570592779264 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54392716 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚422852972645376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚553 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚518323097319280 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚530380482551296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36297762 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18097419 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6571863483904 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159684608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚530455604238848 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚530455604238848 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7844900388164115e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122552488 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122552488 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚523883723236352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚422912643768320 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚518396540441120 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚530455245926912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69222990 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54399971 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69212833 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6571522690560 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚416138747015168 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69212833 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚523883723236352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6571522690560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚524956224192848 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53329496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6773921177600 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6571522690560 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚416138747015168 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69212833 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚523883723236352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6571522690560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚524956224192848 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53329496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6773921177600 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18097419 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6571863483904 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159684608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚530455604238848 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚530455604238848 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7844900388164115e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122552488 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122552488 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚523883723236352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚422912643768320 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚518396540441120 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚530455245926912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69222990 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54399971 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69212833 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚527519137440256 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6617461387264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚300 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6821069190144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53697094 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚298 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚557 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚425844590516736 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54773471 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36551812 % _timestamp‚1.7844900624266195e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚534136240522752 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6617120601088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚528599617076140 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6605215632860 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69708703 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69698560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚534136240522752 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 trainer/global_step‚6899 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6617461387264 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚419023662885376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚534136598827520 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123405799 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚425844756499968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6820927631360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53697094 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69698560 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚528599974038822 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6605555556182 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚527519119921664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚521994401443280 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚419023687309824 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚521994418482640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53697096 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚527519137440256 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 123395654 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚527519119921664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54773771 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 JDevicÁˆµÂ€eStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69708703 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18221959 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18221661QšÄî'žèO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69708703 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18221959 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18221661 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 _runtime‚3539.001467616 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚534136598827520 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚527519137440256 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6617461387264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚300 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6821069190144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53697094 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚298 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚557 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚425844590516736 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54773471 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36551812 % _timestamp‚1.7844900624266195e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚534136240522752 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6617120601088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚528599617076140 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6605215632860 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69708703 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69698560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚534136240522752 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 trainer/global_step‚6899 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6617461387264 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚419023662885376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚534136598827520 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123405799 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚425844756499968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6820927631360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53697094 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69698560 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚528599974038822 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6605555556182 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚527519119921664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚521994401443280 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚419023687309824 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚521994418482640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53697096 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚527519137440256 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 123395654 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚527519119921664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54773771­jª?2Ÿ- _wandbruntime‚3539  _runtime‚3539 «Kó& íM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53704598 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6606483779530 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6618050512384 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36556893 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚528673971381856 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69708435 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 % _timestamp‚1.7844900627852805e+09 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚425904423686656 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6618050512384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53704596 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123423190 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚419082563642368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69708435 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6618391305728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123413031 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚527593311690752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54781109 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚522067844565120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53704596 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚419082539217920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚522067827525760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚527593329209344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚528674328344650 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53704598 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54781403 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69718592 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚534211720515072 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚425904581274112 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18224510 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6821884468736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6606143856096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69718592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚527593329209344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133163008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚534211362203136 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123423190 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 _runtime‚3539.360446932 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36556891 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6822017631744 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6618391305728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18224218 _step‚558 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚425904581274112 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18224510 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6821884468736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6606143856096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69718592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚527593329209344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133163008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚534211362203136 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123423190 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 _runtime‚3539.360446932 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36556891 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6822017631744 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6618391305728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18224218 _step‚558 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6618050512384 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36556893 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚528673971381856 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69708435 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 % _timestamp‚1.7844900627852805e+09 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚425904423686656 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6618050512384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53704596 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123423190 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚419082563642368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69708435 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6618391305728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123413031 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚527593311690752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54781109 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚522067844565120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53704596 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚419082539217920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚522067827525760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚527593329209344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚528674328344650 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53704598 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54781403 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69718592 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935®.<å32¢- _wandbruntime‚3539  _runtime‚3539Í/v £„ 0loss/FRt1/assignment_loss‚0.8065537012353236 2loss/FBt1/assignment_loss‚0.012577585448841806 /loss/FRt2/detection_loss‚0.5326955416842905 1loss/SRqqt2/detection_loss‚1.0230330353521782 %loss/total_loss‚8.346962902485483 _step‚559 2loss/SRqqt1/assignment_loss‚0.5077657443028334 3loss/FBt2/assignment_loss‚0.0007574271685595153 /loss/FRt1/detection_loss‚1.3482381521629132 1loss/SRqqt1/detection_loss‚0.5138951749375467 0loss/FBt1/detection_loss‚0.49214386075909716 /loss/FBt2/detection_loss‚0.7195165322090571 % _timestamp‚1.7844900627862308e+09 /loss/FRt2/assignment_loss‚1.887364667679187 2loss/SRqqt2/assignment_loss‚0.5024214795456551 epoch‚8 trainer/global_step‚6899 _runtime‚3539.360959713žEÖ¤‰ /loss/FBt2/detection_loss‚0.7195165322090571 % _timestamp‚1.7844900627862308e+09 /loss/FRt2/assignment_loss‚1.887364667679187 2loss/SRqqt2/assignment_loss‚0.5024214795456551 epoch‚8 trainer/global_step‚6899 _runtime‚3539.360959713 0loss/FRt1/assignment_loss‚0.8065537012353236 2loss/FBt1/assignment_loss‚0.012577585448841806 /loss/FRt2/detection_loss‚0.5326955416842905 1loss/SRqqt2/detection_loss‚1.0230330353521782 %loss/total_loss‚8.346962902485483 _step‚559 2loss/SRqqt1/assignment_loss‚0.5077657443028334 3loss/FBt2/assignment_loss‚0.0007574271685595153 /loss/FRt1/detection_loss‚1.3482381521629132 1loss/SRqqt1/detection_loss‚0.5138951749375467 0loss/FBt1/detection_loss‚0.49214386075909716¯} (R륂(:Ó ÏÐôÒ¯ðÀË disk.dm-1.out‚ 5592.01953125& disk.nvme0c0n1.out‚34159.615234375" proc.memory.rssMB‚ 2841.2421875) proc.memory.percent‚17.34156608581543) disk./.usagePercent‚76.67119644729074 disk.nvme0nf~Ös1.in‚ 4.3359375$ disk.nvme0n1.out‚34159.615234375# disk.dm-0.out‚38335.30517578125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375( proc.memory.availableMB‚ 493.57421875 cpu‚26.096984701031012 network.sent‚ 9.359473e+06 network.recv‚961578' disk.nvme1c1n1.out‚9971.26708984375% disk.nvme1n1.out‚9971.26708984375 disk.md0.in‚0# memory_percent‚96.9874620437622% disk./.usageGB‚1177.0945434570312 disk.md0.out‚ 94.7578125 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0ILƒ_ž¦‚(:“ ÏÐôÒìëèË gpu.0.gpu‚70.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚269.345' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.78166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä…3c§‚(:Ø ÞÐôÒõÁÒË( disk.nvme0c0n1.out‚34161.33544921875 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375! disk.dm-0.out‚38337.087890625 disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚34161.33544921875 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚9972.09423828125 network.recv‚965786) disk./.usagePercent‚76.67122377943204 disk.nvme1n1.in‚0$ memory_percent‚96.98624610900879 network.sent‚ 9.423644e+06 disk.md0.out‚ 95.01171875 disk.dm-1.in‚0! disk.dm-1.out‚5592.2138671875 disk.rbd0.out‚ 0.4697265625% disk.nvme1n1.out‚9972.09423828125' proc.memory.availableMB‚ 493.7734375 proc.cpu.threads‚17% disk./.usageGB‚1177.0949630737305 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375" proc.memory.rssMB‚ 2841.2421875) proc.memory.percent‚17.34156608581543 cpu‚26.095524180808763ט«*ž¨‚(:“ ÞÐôÒð݇Ì gpu.0.gpu‚91.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚237.587' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚79.19566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ôo2©- _wandbruntime‚3562  _runtime‚3562¹Ö/ɪà ' lr-AdamW/pg1‚0.00020022108810884123 ' lr-AdamW/pg2‚0.00020022108810884123 trainer/global_step‚6949 % _timestamp‚1.7844900864226367e+09 _runtime‚3562.997601686 _step‚560{±^SÎ«È ' lr-AdamW/pg1‚0.00020022108810884123 ' lr-AdamW/pg2‚0.00020022108810884123 trainer/global_step‚6949 % _timestamp‚1.7844900864226367e+09 _runtime‚3562.997601686 _step‚560°HŽÄR2¬- _wandbruntime‚3562  _runtime‚3562vEBºé'­ãO cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6868997156864 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6663989209088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18347321 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 _step‚561 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70204305 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18347611 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6651998621190 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36805860 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6868874472448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚532317721227830 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚531228708376064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚421967503937024 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚537892356798976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚537892715103744 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚428836501093888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55153473 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55153181 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚428836353985024 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6663648422912 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54072194 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70194162 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 trainer/global_step‚6949 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚537892356798976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚531228708376064 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124276501 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 _runtime‚3562.998313497 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54072194 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36805862 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70204305 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚531228725894656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚421967479512576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚532317364265148 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720Àb4wî'®èO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚537892715103744 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚428836501093888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55153473 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55153181 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚428836353985024 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6663648422912 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54072194 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70194162 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 trainer/global_step‚6949 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚537892356798976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚531228708376064 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124276501 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 _runtime‚3562.998313497 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54072194 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36805862 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70204305 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚531228725894656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚421967479512576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚532317364265148 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6868997156864 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6663989209088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18347321 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 _step‚561 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70204305 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18347611 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6651998621190 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36805860 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6868874472448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚532317721227830 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚531228708376064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚421967503937024 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚537892356798976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360±ìò= 2¯- _wandbruntime‚3563  _runtime‚3563ô7P–$°íM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚531302917663744 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚428896192453120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6664578334208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6664919127552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18349807 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚531302917663744 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55160748 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18350077 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚562 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70204037 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6869961382400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124293892 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55161020 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54079698 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚525739131649760 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 % _timestamp‚1.7844900867822964e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 124283733 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚531302900145152 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70204037 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚525739148689120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚537967836791296 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70214194 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6664578334208 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70214194 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚537967478479360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 124283733 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6652586921104 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 trainer/global_step‚6949 _runtime‚3563.357426431 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6664919127552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚531302900145152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36810941 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚532392075533658 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.curreì¦^ü]nt‚ 149198848 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚422026355845120 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6652926844538 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280»Lšóø&±òM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18350077 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚562 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70204037 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6869961382400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124293892 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55161020 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54079698 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚525739131649760 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 % _timestamp‚1.7844900867822964e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 124283733 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚531302900145152 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70204037 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚525739148689120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚537967836791296 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70214194 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6664578334208 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70214194 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚537967478479360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 124283733 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6652586921104 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 trainer/global_step‚6949 _runtime‚3563.357426431 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6664919127552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚531302900145152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36810941 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚532392075533658 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚422026355845120 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6652926844538 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚531302917663744 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚428896192453120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6664578334208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6664919127552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18349807 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚531302917663744 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55160748²ÚÎì2²- _wandbruntime‚3563  _runtime‚3563µs™\³€ trainer/global_step‚6949 2loss/SRqqt1/assignment_loss‚0.5797373347446751 1loss/FBt2/assignment_loss‚0.16393449660464074 /loss/FBt1/detection_loss‚0.5070004257803353 %loss/total_loss‚9.218510060990845 _step‚563 0loss/FRt2/assignment_loss‚1.8721455280384962 2loss/SRqqt2/assignment_loss‚0.7427112032870685 1loss/SRqqt2/detection_loss‚0.9122225715387189 epoch‚8 % _timestamp‚1.7844900867832363e+09 _runtime‚3563.357928402 2loss/FBt1/assignment_loss‚0.006706977890615589 /loss/FRt1/detection_loss‚1.5081430455007487 -loss/FBt2/detection_loss‚0.74426850177009 0loss/FRt1/assignment_loss‚0.9985647527376811 /loss/FRt2/detection_loss‚0.5684922700171952 1loss/SRqqt1/detection_loss‚0.6145829530805802’ච´… 2loss/SRqqt1/assignment_loss‚0.5797373347446751 1loss/FBt2/assignment_loss‚0.16393449660464074 /loss/FBt1/detection_loss‚0.5070004257803353 %loss/total_loss‚9.218510060990845 _step‚563 0loss/FRt2/assignment_loss‚1.8721455280384962 2loss/SRqqt2/assignment_loss‚0.7427112032870685 1loss/SRqqt2/detection_loss‚0.9122225715387189 epoch‚8 % _timestamp‚1.7844900867832363e+09 _runtime‚3563.357928402 2loss/FBt1/assignment_loss‚0.006706977890615589 /loss/FRt1/detection_loss‚1.5081430455007487 -loss/FBt2/detection_loss‚0.74426850177009 0loss/FRt1/assignment_loss‚0.9985647527376811 /loss/FRt2/detection_loss‚0.5684922700171952 1loss/SRqqt1/detection_loss‚0.6145829530805802 trainer/global_step‚6949³¡ÓFbµ‚(:× íÐôÒû¬åË% disk./.usageGB‚1176.7575225830078 disk.rbd0.out‚ 0.4697265625' disk.nvme1c1n1.out‚9972.98876953125 disk.nvme1n1.in‚0 disk.dm-0.in‚ 4.3359375) proc.memory.percent‚17.34156608581543% disk.nvme1n1.out‚9972.98876953125 network.sent‚ 9.426608e+06 network.recv‚968877 disk.rbd0.in‚0( disk.nvme0c0n1.out‚34162.28466796875& disk.nvme0n1.out‚34162.28466796875 cpu‚26.094564085482325 disk.nvme1c1n1.in‚0 disk.md0.out‚ 95.41796875$ memory_percent‚96.98667526245117" proc.memory.rssMB‚ 2841.2421875 proc.cpu.threads‚17) disk./.usagePercent‚76.64924426529932 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 disk.md0.in‚0! disk.dm-0.out‚38337.966796875 disk.dm-1.in‚0! disk.dm-1.out‚5592.3193359375& proc.memory.availableMB‚ 493.703125R¤î%ž¶‚(:“ íÐôÒ“ÆÆÌ gpu.0.gpu‚51.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚263.171' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.72366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜ä;Ta·‚(:Ö üÐôÒ駒Ì& disk.nvme0n1.out‚34164.80419921875( proc.memory.availableMB‚ 493.49609375) proc.memory.percent‚17.34156608581543 network.sent‚ 9.491151e+06 disk.nvme1c1n1.in‚0 disk.md0.in‚0# disk.dm-0.out‚38340.47900390625! disk.dm-1.out‚5592.5009765625 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375% disk.nvme1c1n1.out‚9973.865234375 disk.nvme1n1.in‚0 disk.md0.out‚ 95.74609375 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0) disk./.usagePercent‚76.64927582149882 disk.nvme0n1.in‚ 4.3359375# disk.nvme1n1.out‚9973.865234375$ memory_percent‚96.98793888092041" proc.memory.rssMB‚ 2841.2421875 cpu‚26.09569283673307 proc.cpu.threads‚17 network.recv‚973213% disk./.usageGB‚1176.7580070495605( disk.nvme0c0n1.out‚34164.80419921875ßL5Ξ¸‚(:“ üÐôÒîŒÌ gpu.0.gpu‚57.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.492' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.83066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·£\ï2¹- _wandbruntime‚3588  _runtime‚3588Få›%ɺà _runtime‚3588.543673921 _step‚564 ' lr-AdamW/pg1‚0.00020001909650091532 ' lr-AdamW/pg2‚0.00020001909650091532 trainer/global_step‚6999 % _timestamp‚1.7844901119686775e+09/À·Î»È trainer/global_step‚6999 % _timestamp‚1.7844901119686775e+09 _runtime‚3588.543673921 _step‚564 ' lr-AdamW/pg1‚0.00020001909650091532 ' lr-AdamW/pg2‚0.00020001909650091532´+:÷02¼- _wandbruntime‚3588  _runtime‚3588|²ªòé'½ãO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 125137058 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18474251 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55533878 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚431828323571200 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚541648473075200 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54447294 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚536035111454156 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6916865642496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6917003006976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54447296 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6698101762876 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 137364480 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54447296 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70699907 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6710517030912 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70689764 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚541648473075200 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚541648831379968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚534938314349056 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚424911320564224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚541648831379968 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70699907 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37059910 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 125137058 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18474251 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55533878 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚431828323571200 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚541648473075200 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54447294 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚536035111454156 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6916865642496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6917003006976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54447296 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6698101762876 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 137364480 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54447296 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70699907 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6710517030912 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70689764 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚541648473075200 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚541648831379968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚534938314349056 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚424911320564224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚541648831379968 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70699907 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37059910 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 trainer/global_step‚6999 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6711106156032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18476413 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚431887998636544 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚535012488599552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37064991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6711446949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54454798 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6711106156032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125154435 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚529410435773760 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚535012506118144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54454798 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125164594 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54454796 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6699029986112 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7844901123440087e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚541723953067520 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70709796 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55541680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6917826164224 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70699639 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55541404 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6917957230080 _step‚566 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚431888154126848 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54454796 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚535012488599552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚541723594755584 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚536109822722666ïe@7ø&ÁòM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚535012488599552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37064991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6711446949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54454798 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6711106156032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125154435 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚529410435773760 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚535012506118144 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54454798 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125164594 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54454796 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6699029986112 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7844901123440087e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚541723953067520 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70709796 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55541680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6917826164224 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70699639 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55541404 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6917957230080 _step‚566 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚431888154126848 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54454796 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚535012488599552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚541723594755584 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚536109822722666 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 trainer/global_step‚6999 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6711106156032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18476413 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚431887998636544 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0¶, ÿy2Â- _wandbruntime‚3588  _runtime‚3588‰Ã{Ãþ _step‚567 0loss/FRt1/assignment_loss‚0.8160621583023492 epoch‚8 /loss/FBt2/detection_loss‚1.0842445407884953 2loss/SRqqt1/assignment_loss‚0.5573988839266045 2loss/SRqqt2/assignment_loss‚0.6394988246593574 1loss/FBt1/assignment_loss‚0.03919990104194574 1loss/SRqqt1/detection_loss‚0.6893705259232471 1loss/SRqqt2/detection_loss‚1.0061792615664804 /loss/FBt1/detection_loss‚0.6307538419452338 /loss/FRt2/assignment_loss‚1.884129027053553 .loss/FRt2/detection_loss‚0.499180358796527 %loss/total_loss‚9.229416719661643 1loss/FBt2/assignment_loss‚0.01317586990720585 /loss/FRt1/detection_loss‚1.3702235257506425 trainer/global_step‚6999 $ _timestamp‚1.784490112345097e+09 _runtime‚3588.919792936>^§° ă 1loss/FBt2/assignment_loss‚0.01317586990720585 /loss/FRt1/detection_loss‚1.3702235257506425 trainer/global_step‚6999 $ _timestamp‚1.784490112345097e+09 _runtime‚3588.919792936 _step‚567 0loss/FRt1/assignment_loss‚0.8160621583023492 epoch‚8 /loss/FBt2/detection_loss‚1.0842445407884953 2loss/SRqqt1/assignment_loss‚0.5573988839266045 2loss/SRqqt2/assignment_loss‚0.6394988246593574 1loss/FBt1/assignment_loss‚0.03919990104194574 1loss/SRqqt1/detection_loss‚0.6893705259232471 1loss/SRqqt2/detection_loss‚1.0061792615664804 /loss/FBt1/detection_loss‚0.6307538419452338 /loss/FRt2/assignment_loss‚1.884129027053553 .loss/FRt2/detection_loss‚0.499180358796527 %loss/total_loss‚9.229416719661643·õ\ê YÅ‚(:Î ‹ÑôÒ´Å¾Ì disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚96.125 disk.rbd0.in‚0& disk.nvme0c0n1.out‚34165.689453125" proc.memory.rssMB‚ 2841.2421875 proc.cpu.threads‚17 network.recv‚983720% disk./.usageGB‚1176.7580680847168 disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme1c1n1.in‚0# disk.dm-0.out‚38341.02587890625) proc.memory.percent‚17.34156608581543 cpu‚26.094114881550016 network.sent‚ 9.51067e+06! disk.dm-1.out‚5592.5634765625$ disk.nvme0n1.out‚34165.689453125% disk.nvme1n1.out‚9974.40673828125 disk.dm-0.in‚ 4.3359375$ memory_percent‚96.98824882507324' proc.memory.availableMB‚ 493.4453125' disk.nvme1c1n1.out‚9974.40673828125' disk./.usagePercent‚76.649279797083/0¬<Æ‚(:’ ‹ÑôÒà¾Ì gpu.0.gpu‚56.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚247.84' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚82.61333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øY S2Ç- _wandbruntime‚3613  _runtime‚3613…Š}ÈÈ $ _timestamp‚1.784490137222688e+09 _runtime‚3613.797674559 _step‚568 ' lr-AdamW/pg1‚0.00019981560578762617 ' lr-AdamW/pg2‚0.00019981560578762617 trainer/global_step‚7049p´ì§ÍÉÇ ' lr-AdamW/pg1‚0.00019981560578762617 ' lr-AdamW/pg2‚0.00019981560578762617 trainer/global_step‚7049 $ _timestamp‚1.784490137222688e+09 _runtime‚3613.797674559 _step‚568¸z­l-2Ê- _wandbruntime‚3613  _runtime‚3613©Ú{ðËãO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚434820080007168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159691776 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 135267328 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37313960 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6964942815744 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6964807548416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚545404947656192 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚545404589351424 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚545404947656192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6744544827884 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚533008313815280 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54822394 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18600741 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚427855112766976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0÷aRfî'ÌèO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71195509 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚539753215605846 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 trainer/global_step‚7049 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 126017905 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71185366 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚545404589351424 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚539752858643164 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18600741 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚427855112766976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚434820080007168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159691776 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 135267328 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37313960 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6964942815744 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6964807548416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚545404947656192 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚545404589351424 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚545404947656192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6744544827884 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚533008313815280 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54822394 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚427913989099520 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 trainer/global_step‚7049 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚545479711031808 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55922454 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚427914013523968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71205398 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18603123 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚538722077053952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18603411 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71205398 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚434879749401600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 _step‚570 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161781760 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126035296 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 126035296 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6965760302080 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚545480069343744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 126025137 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚545480069343744 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6757633977856 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 _runtime‚3614.175060573 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54829898 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚538722077053952 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚539827569911674 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55922164 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚434879911183360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360e½kÛc!ÏñM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚545480069343744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 126025137 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚545480069343744 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6757633977856 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 _runtime‚3614.175060573 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54829898 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚538722077053952 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚539827569911674 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55922164 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚434879911183360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71195241 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 126025137 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6745473051120 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54829898 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37319043 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚538722094572544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚427913989099520 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 trainer/global_step‚7049 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚545479711031808 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55922454 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚427914013523968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71205398 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18603123 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚538722077053952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18603411 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 IyŽC”VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71205398 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚434879749401600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 _step‚570 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161781760 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126035296 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 126035296 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6965760302080ºNwÕ®2Ð- _wandbruntime‚3614  _runtime‚3614‰QÑÿ 1loss/SRqqt1/detection_loss‚0.5942822800793583 0loss/FBt1/detection_loss‚0.41780152994959446 /loss/FBt2/detection_loss‚0.6182225333149767 0loss/FRt2/assignment_loss‚1.9912897793094764 2loss/SRqqt1/assignment_loss‚0.5607165269040986 /loss/FRt2/detection_loss‚0.4941165615007163 trainer/global_step‚7049 # _timestamp‚1.78449013760095e+09 _runtime‚3614.175772154 /loss/SRqqt2/detection_loss‚1.15319660468027 0loss/FRt1/assignment_loss‚0.8884559044955482 epoch‚8 _step‚571 %loss/total_loss‚8.716122613861332 2loss/SRqqt2/assignment_loss‚0.5389582572078022 2loss/FBt1/assignment_loss‚0.022184152542260967 2loss/FBt2/assignment_loss‚0.007081714050990905 .loss/FRt1/detection_loss‚1.429816769826239Þj«… Ò„ %loss/total_loss‚8.716122613861332 2loss/SRqqt2/assignment_loss‚0.5389582572078022 2loss/FBt1/assignment_loss‚0.022184152542260967 2loss/FBt2/assignment_loss‚0.007081714050990905 .loss/FRt1/detection_loss‚1.429816769826239 1loss/SRqqt1/detection_loss‚0.5942822800793583 0loss/FBt1/detection_loss‚0.41780152994959446 /loss/FBt2/detection_loss‚0.6182225333149767 0loss/FRt2/assignment_loss‚1.9912897793094764 2loss/SRqqt1/assignment_loss‚0.5607165269040986 /loss/FRt2/detection_loss‚0.4941165615007163 trainer/global_step‚7049 # _timestamp‚1.78449013760095e+09 _runtime‚3614.175772154 /loss/SRqqt2/detection_loss‚1.15319660468027 0loss/FRt1/assignment_loss‚0.8884559044955482 epoch‚8 _step‚571»žÛGl^Ó‚(:Ó šÑôÒÕÇÌ proc.cpu.threads‚17$ disk.nvme0n1.out‚34166.435546875& disk.nvme1c1n1.out‚9974.8857421875 disk.rbd0.out‚ 0.4697265625# memory_percent‚96.9881534576416 network.recv‚988200) disk./.usagePercent‚76.64928103945307 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚9974.8857421875 disk.rbd0.in‚0' proc.memory.availableMB‚ 493.4609375 network.sent‚ 9.573058e+06% disk./.usageGB‚1176.7580871582031& disk.nvme0c0n1.out‚34166.435546875 disk.nvme1c1n1.in‚0 disk.md0.out‚ 96.39453125" proc.memory.rssMB‚ 2841.2421875) proc.memory.percent‚17.34156608581543 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚38341.58447265625 disk.dm-1.in‚0" disk.dm-1.out‚5592.65966796875 cpu‚26.09256143246208°‹Ï'“Ô‚(:ˆ šÑôÒ¼Å»Ì gpu.0.gpu‚90.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚242.787' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚80.929 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðzŽ·dÕ‚(:Ù ©ÑôÒ–¦¶Ë disk.dm-0.in‚ 4.3359375" disk.dm-0.out‚38343.8525390625" disk.dm-1.out‚5592.76513671875 proc.cpu.threads‚17 network.recv‚992672& disk.nvme0n1.out‚34168.73095703125 disk.nvme1n1.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625( proc.memory.availableMB‚ 493.63671875 network.sent‚ 9.622963e+06 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 disk.md0.out‚ 96.72265625 disk.rbd0.in‚0$ memory_percent‚96.98708057403564% disk./.usageGB‚1176.7587356567383( disk.nvme0c0n1.out‚34168.73095703125 disk.nvme1c1n1.in‚0$ disk.nvme1n1.out‚9975.6748046875" proc.memory.rssMB‚ 2841.2421875) proc.memory.percent‚17.34156608581543 cpu‚26.093698613541083) disk./.usagePercent‚76.64932328003506& disk.nvme1c1n1.out‚9975.6748046875 disk.md0.in‚0ëËQýžÖ‚(:“ ©ÑôÒŒâ“Ì gpu.0.gpu‚82.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚184.936' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚61.64533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š¤Í¸2×- _wandbruntime‚3638  _runtime‚3638¤L³dÇØÁ & lr-AdamW/pg1‚0.0001996106192282906 & lr-AdamW/pg2‚0.0001996106192282906 trainer/global_step‚7099 % _timestamp‚1.7844901621920767e+09 _runtime‚3638.767039219 _step‚572+FP&ÌÙÆ & lr-AdamW/pg2‚0.0001996106192282906 trainer/global_step‚7099 % _timestamp‚1.7844901621920767e+09 _runtime‚3638.767039219 _step‚572 & lr-AdamW/pg1‚0.0001996106192282906¼PÆ2Ú- _wandbruntime‚3638  _runtime‚3638»`è'ÛâO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7012790568960 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71680968 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56294606 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 _step‚573 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6803572674560 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 trainer/global_step‚7099 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71691111 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6791327816214 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18726596 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚543470605832172 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6803231888384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71680968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚536679617939280 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55197494 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55197494 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 126888607 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18726875 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚549160705627648 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚549160705627648 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚542357491257856 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71691111 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚437811719963136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚549161063932416 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚430798929394176 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚549160705627648 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚549160705627648 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚542357491257856 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71691111 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚437811719963136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚549161063932416 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚430798929394176 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 126888607 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834½#š‹Î2Ý- _wandbruntime‚3639  _runtime‚3639ýà¹ÊÞíM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6804161799680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚536753044021760 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55204998 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚430857830151168 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7013743707648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7099 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56302477 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56302191 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 126895839 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37573093 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚549235827308032 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55204998 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71701000 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚542431683026944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37573091 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7013876870656 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚536753061061120 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126905998 % _timestamp‚1.7844901625498817e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71690843 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚542431665508352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55204996 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚543545317100682 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚542431683026944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚£ápz)260 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 126895839 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 _step‚574 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18729100 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚549236185619968 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6804502593024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6804502593024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18729384 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚549236185619968 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71701000 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71690843 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚542431665508352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55204996 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2,ó‘ø&ßòM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55204998 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71701000 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚542431683026944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37573091 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7013876870656 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚536753061061120 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126905998 % _timestamp‚1.7844901625498817e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71690843 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚542431665508352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55204996 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚543545317100682 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚542431683026944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 126895839 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 _step‚574 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18729100 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚549236185619968 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6804502593024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6804502593024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18729384 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚549236185619968 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71701000 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71690843 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚542431665508352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55204996 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6804161799680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚536753044021760 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55204998 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚430857830151168 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7013743707648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7099 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56302477 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56302191 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 126895839 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37573093 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚549235827308032 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0¾g vÂ2à- _wandbruntime‚3639  _runtime‚3639@ªá€ trainer/global_step‚7099 % _timestamp‚1.7844901625512018e+09 1loss/FBt2/assignment_loss‚0.15639667539139002 1loss/SRqqt2/detection_loss‚0.9272944286501237 /loss/FBt2/detection_loss‚0.8342771670260631 epoch‚8 .loss/FRt2/assignment_loss‚2.03775126727358 3loss/SRqqt1/assignment_loss‚0.49423116100254894 2loss/SRqqt2/assignment_loss‚0.6281480158777065 3loss/FBt1/assignment_loss‚0.0003991981359645123 /loss/FBt1/detection_loss‚0.4812193644503432 %loss/total_loss‚8.835812065378626 _runtime‚3639.125867007 _step‚575 /loss/FRt1/assignment_loss‚0.814774307631707 /loss/FRt1/detection_loss‚1.3385782127680128 .loss/FRt2/detection_loss‚0.515150512826509 1loss/SRqqt1/detection_loss‚0.6075917543446764?²Èƒ â… .loss/FRt2/detection_loss‚0.515150512826509 1loss/SRqqt1/detection_loss‚0.6075917543446764 trainer/global_step‚7099 % _timestamp‚1.7844901625512018e+09 1loss/FBt2/assignment_loss‚0.15639667539139002 1loss/SRqqt2/detection_loss‚0.9272944286501237 /loss/FBt2/detection_loss‚0.8342771670260631 epoch‚8 .loss/FRt2/assignment_loss‚2.03775126727358 3loss/SRqqt1/assignment_loss‚0.49423116100254894 2loss/SRqqt2/assignment_loss‚0.6281480158777065 3loss/FBt1/assignment_loss‚0.0003991981359645123 /loss/FBt1/detection_loss‚0.4812193644503432 %loss/total_loss‚8.835812065378626 _runtime‚3639.125867007 _step‚575 /loss/FRt1/assignment_loss‚0.814774307631707 /loss/FRt1/detection_loss‚1.3385782127680128¿BG°_ã‚(:Ô ¸ÑôÒ˜ÿùË disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625& disk.nvme0n1.out‚34169.98095703125! disk.nvme1n1.out‚ 9976.3046875" proc.memory.rssMB‚ 2841.2421875) proc.memory.percent‚17.34156608581543 network.recv‚ 1.00301e+06" disk.dm-0.out‚38344.8564453125( disk.nvme0c0n1.out‚34169.98095703125 disk.md0.out‚ 97.046875 cpu‚26.092964948979137 disk.dm-0.in‚ 4.3359375 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0$ memory_percent‚96.98731899261475 proc.cpu.threads‚17$ disk./.usageGB‚1176.758602142334" disk.dm-1.out‚5592.94580078125 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375# disk.nvme1c1n1.out‚ 9976.3046875 disk.nvme1n1.in‚0 disk.md0.in‚0( proc.memory.availableMB‚ 493.59765625 network.sent‚ 9.64502e+06) disk./.usagePercent‚76.64931458344465» žä‚(:“ ¸ÑôÒÖ±Ì gpu.0.gpu‚75.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚230.833' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚76.94433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºzR¾kå‚(:à ÇÑôÒ•ÓÌË) proc.memory.percent‚17.34156608581543 network.recv‚ 1.007476e+06 disk.nvme0c0n1.in‚ 4.3359375( disk.nvme0c0n1.out‚34171.98486328125& disk.nvme0n1.out‚34171.98486328125 disk.dm-1.in‚0' proc.memory.availableMB‚ 493.5859375 cpu‚26.091774074385828 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0# disk.dm-0.out‚38346.64208984375 disk.rbd0.out‚ 0.4697265625" proc.memory.rssMB‚ 2841.2421875) disk./.usagePercent‚76.65328345883458 disk.nvme0n1.in‚ 4.3359375 disk.nvme1n1.in‚0% disk.nvme1n1.out‚9976.76904296875 disk.dm-0.in‚ 4.3359375 disk.rbd0.in‚0$ memory_percent‚96.98739051818848 network.sent‚ 9.69356e+06% disk./.usageGB‚1176.8195343017578' disk.nvme1c1n1.out‚9976.76904296875 disk.md0.in‚0 disk.md0.out‚ 97.33984375" disk.dm-1.out‚5592.98876953125¬®eÈžæ‚(:“ ÇÑôÒʸ˜Ì gpu.0.gpu‚90.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚242.294' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚80.76466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìĨà2ç- _wandbruntime‚3664  _runtime‚36643BXaÉèà ' lr-AdamW/pg1‚0.00019940414010618444 ' lr-AdamW/pg2‚0.00019940414010618444 trainer/global_step‚7149 % _timestamp‚1.7844901876152136e+09 _runtime‚3664.190215077 _step‚576­X†ÌÎéÈ ' lr-AdamW/pg1‚0.00019940414010618444 ' lr-AdamW/pg2‚0.00019940414010618444 trainer/global_step‚7149 % _timestamp‚1.7844901876152136e+09 _runtime‚3664.190215077 _step‚576Àn0äž2ê- _wandbruntime‚3664  _runtime‚3664Lß<@è'ëâO P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72186713 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18853697 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚552917180208640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37822060 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚440803673778176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚552917180208640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚440803522475008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56675476 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6850100496384 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 _step‚577 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚546067062193664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7060776453632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55572596 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55572594 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6849759710208 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6849759710208 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6850100496384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6837430957900 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127759309 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18853416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6837770881222 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37822062 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚546067079712256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55572596 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚546067062193664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚7149 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚546067079712256 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72176570 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚552916821903872 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7060903332352 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72176570 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚540350922063280 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127749164 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚552916821903872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚433742770445824 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚540350939102640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚547188709983862 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚547188353021180 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 % _timestamp‚1.7844901876163008e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚433742746021376CÜûJìçO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72176570 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚552916821903872 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7060903332352 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72176570 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚540350922063280 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127749164 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚552916821903872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚433742770445824 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚540350939102640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚547188709983862 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚547188353021180 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 % _timestamp‚1.7844901876163008e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚433742746021376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72186713 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18853697 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚552917180208640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37822060 ^JDeviceStatsMonitor.on_train_ Üa%£batch_start/inactive_split_bytes.all.allocated‚440803673778176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚552917180208640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚440803522475008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56675476 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6850100496384 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 _step‚577 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚546067062193664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7060776453632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55572596 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55572594 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6849759710208 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6849759710208 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6850100496384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6837430957900 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127759309 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18853416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6837770881222 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37822062 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚546067079712256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55572596 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚546067062193664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚7149 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚546067079712256Á•ñŽ„2í- _wandbruntime‚3664  _runtime‚3664k¶†;ó&îíM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7061736303616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7061863175168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55580096 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127776700 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72196602 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55580096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6850689621504 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55580098 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚540424365185120 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚552992301896192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚552991943584256 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72186445 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37827141 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚547262707326896 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6851030414848 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72196602 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚440863358657536 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 _runtime‚3664.567900005 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚546141271481344 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚546141271481344 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚546141253962752 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚433801646778368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127776700 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚546141253962752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6838699104570 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56683092 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18856239 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72186445 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37827143 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6850689621504 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚552992301896192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 _step‚578 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127766541 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55580098 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56683382 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127766541ä8#Áø&ïòM TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6851030414848 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72196602 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚440863358657536 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 _runtime‚3664.567900005 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚546141271481344 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚546141271481344 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚546141253962752 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚433801646778368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127776700 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚546141253962752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6838699104570 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56683092 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18856239 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72186445 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37827143 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6850689621504 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚552992301896192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 _step‚578 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127766541 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55580098 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56683382 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127766541 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7061736303616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7061863175168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55580096 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127776700 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72196602 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55580096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6850689621504 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55580098 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚540424365185120 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚552992301896192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚552991943584256 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72186445 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37827141 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚547262707326896 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087ÂjÙ}b2ð- _wandbruntime‚3664  _runtime‚3664X0ï$ ñ‡ 2loss/SRqqt2/assignment_loss‚0.5540211898521462 4loss/FBt1/assignment_loss‚0.00024968828347613663 /loss/FRt2/detection_loss‚0.5255951584355568 trainer/global_step‚7149 _step‚579 4loss/FBt2/assignment_loss‚0.00021871027642912918 /loss/FRt1/detection_loss‚1.3751473101474536 0loss/FBt1/detection_loss‚0.35414951036039455 %loss/total_loss‚8.147347253367787 epoch‚8 0loss/FRt1/assignment_loss‚0.7851650312377958 0loss/FRt2/assignment_loss‚1.9347717712875996 1loss/SRqqt1/assignment_loss‚0.630654019175489 1loss/SRqqt2/detection_loss‚0.8776616035380734 /loss/FBt2/detection_loss‚0.5017578322943803 % _timestamp‚1.7844901879938147e+09 1loss/SRqqt1/detection_loss‚0.6079554284789921 _runtime‚3664.568486087p nÁòŒ 4loss/FBt2/assignment_loss‚0.00021871027642912918 /loss/FRt1/detection_loss‚1.3751473101474536 0loss/FBt1/detection_loss‚0.35414951036039455 %loss/total_loss‚8.147347253367787 epoch‚8 0loss/FRt1/assignment_loss‚0.7851650312377958 0loss/FRt2/assignment_loss‚1.9347717712875996 1loss/SRqqt1/assignment_loss‚0.630654019175489 1loss/SRqqt2/detection_loss‚0.8776616035380734 /loss/FBt2/detection_loss‚0.5017578322943803 % _timestamp‚1.7844901879938147e+09 1loss/SRqqt1/detection_loss‚0.6079554284789921 _runtime‚3664.568486087 2loss/SRqqt2/assignment_loss‚0.5540211898521462 4loss/FBt1/assignment_loss‚0.00024968828347613663 /loss/FRt2/detection_loss‚0.5255951584355568 trainer/global_step‚7149 _step‚579Ã’zdó‚(:Ù ÖÑôÒ¼›ˆÌ) proc.memory.percent‚17.34156608581543 disk.md0.out‚ 97.76171875 disk.dm-0.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375' proc.memory.availableMB‚ 493.5546875 network.sent‚ 9.712444e+06) disk./.usagePercent‚76.57129796780858% disk./.usageGB‚1175.5608520507812 disk.rbd0.out‚ 0.4697265625& disk.nvme0n1.out‚34196.74169921875% disk.nvme1c1n1.out‚9977.607421875" proc.memory.rssMB‚ 2841.2421875 network.recv‚ 1.011289e+06 disk.nvme1n1.in‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 4.3359375( disk.nvme0c0n1.out‚34196.74169921875 cpu‚26.091394001605003 proc.cpu.threads‚17# disk.nvme1n1.out‚9977.607421875 disk.md0.in‚0 disk.dm-0.out‚ 38371.0546875" disk.dm-1.out‚5593.26416015625 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0$ memory_percent‚96.98758125305176ÂçAfžô‚(:“ ÖÑôÒžŽ¾Ì gpu.0.gpu‚57.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚120.511' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.17033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Øz82õ- _wandbruntime‚3689  _runtime‚3689>æ«Èö _step‚580 ' lr-AdamW/pg1‚0.00019919617172848994 ' lr-AdamW/pg2‚0.00019919617172848994 trainer/global_step‚7199 $ _timestamp‚1.784490212959106e+09 _runtime‚3689.534058013Â×wéÍ÷Ç ' lr-AdamW/pg1‚0.00019919617172848994 ' lr-AdamW/pg2‚0.00019919617172848994 trainer/global_step‚7199 $ _timestamp‚1.784490212959106e+09 _runtime‚3689.534058013 _step‚580ÄZï6F2ø- _wandbruntime‚3689  _runtime‚3689 w{ä’ ùâO VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38076112 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚436686587073024 _step‚581 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128630011 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6883874022908 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128619866 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚544022226187280 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 120587264 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7108817021440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57057546 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚556672938180096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚550906457172870 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72672172 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚550906100210188 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18981434 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6896287532032 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚443795524681728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38076110 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚549776668166656 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72682315 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚556672938180096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6896287532032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18981164 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚556672938180096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚550906457172870 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72672172 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 177185280 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚550906100210188 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18981434 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6896287532032 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚443795524681728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38076110 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚549776668166656 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72682315 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚556672938180096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6896287532032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18981164 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128630011 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6883874022908 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128619866 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚544022226187280 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 120587264 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7108817021440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57057546 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚544095669309120 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72682047 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚436745463405568 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72682047 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚556748418172416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6897217443328 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚556748059860480 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128647402 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55955198 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7109896097280 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6897558236672 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚549850842417152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 110094336 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 _step‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55955196 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6897558236672 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚544095652269760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚549850859935744 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57064889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 134518784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128637243 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18983698 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18983966 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38081193 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 128647402 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55955198 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 128637243 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚550980454515904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚549850842417152 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72692204 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 % _timestamp‚1.7844902133383276e+09 _runtime‚3689.913453849 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57065159 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128647402 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55955198 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7109896097280 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6897558236672 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚549850842417152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 110094336 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 _step‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55955196 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6897558236672 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚544095652269760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚549850859935744 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 177185280 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57064889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 134518784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128637243 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18983698 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18983966 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38081193 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 128647402 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55955198 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 >n+~ƒ8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 128637243 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚550980454515904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚549850842417152 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72692204 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 % _timestamp‚1.7844902133383276e+09 _runtime‚3689.913453849 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57065159 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚544095669309120 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72682047 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚436745463405568 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72682047 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚556748418172416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935ƲÓ2þ- _wandbruntime‚3689  _runtime‚3689XM°ÿ€ 0loss/FRt1/assignment_loss‚0.9760500413570625 2loss/FBt1/assignment_loss‚0.012470423868388062 /loss/FRt2/detection_loss‚0.5054608898409156 1loss/SRqqt1/detection_loss‚0.6316272913001176 %loss/total_loss‚8.585964573281608 epoch‚8 trainer/global_step‚7199 $ _timestamp‚1.784490213339405e+09 0loss/FRt2/assignment_loss‚2.0059612752724085 2loss/SRqqt1/assignment_loss‚0.5818345918771024 /loss/FRt1/detection_loss‚1.4102919649410972 0loss/FBt1/detection_loss‚0.40204582925294885 /loss/FBt2/detection_loss‚0.5167748604997479 _step‚583 1loss/SRqqt2/assignment_loss‚0.609631549623509 1loss/FBt2/assignment_loss‚0.07949452254649125 1loss/SRqqt2/detection_loss‚0.8543213329018176 _runtime‚ 3689.91451669å¾ø €… 0loss/FRt1/assignment_loss‚0.9760500413570625 2loss/FBt1/assignment_loss‚0.012470423868388062 /loss/FRt2/detection_loss‚0.5054608898409156 1loss/SRqqt1/detection_loss‚0.6316272913001176 %loss/total_loss‚8.585964573281608 epoch‚8 trainer/global_step‚7199 $ _timestamp‚1.784490213339405e+09 0loss/FRt2/assignment_loss‚2.0059612752724085 2loss/SRqqt1/assignment_loss‚0.5818345918771024 /loss/FRt1/detection_loss‚1.4102919649410972 0loss/FBt1/detection_loss‚0.40204582925294885 /loss/FBt2/detection_loss‚0.5167748604997479 _step‚583 1loss/SRqqt2/assignment_loss‚0.609631549623509 1loss/FBt2/assignment_loss‚0.07949452254649125 1loss/SRqqt2/detection_loss‚0.8543213329018176 _runtime‚ 3689.91451669ǵ»Kl‚(:á åÑôÒ„‘âË" proc.memory.rssMB‚ 2841.2421875 cpu‚26.09291816826059) disk./.usagePercent‚76.57129324680237 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0% disk.nvme1n1.out‚9980.75537109375 disk.md0.out‚ 98.08203125 disk.dm-1.in‚0" disk.dm-1.out‚5593.30712890625( disk.nvme0c0n1.out‚34209.47216796875% disk./.usageGB‚1175.5607795715332& disk.nvme0n1.out‚34209.47216796875 disk.dm-0.in‚ 4.3359375$ memory_percent‚96.99232578277588( proc.memory.availableMB‚ 492.77734375) proc.memory.percent‚17.34156608581543 proc.cpu.threads‚17 network.sent‚ 9.760819e+06# disk.dm-0.out‚38386.18798828125 disk.rbd0.out‚ 0.4697265625 network.recv‚ 1.015451e+06 disk.nvme0n1.in‚ 4.3359375' disk.nvme1c1n1.out‚9980.75537109375 disk.md0.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375+Žùž‚‚(:“ åÑôÒŽªÑÌ gpu.0.gpu‚90.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚262.773' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.59100000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u‹Ckcƒ‚(:Ø ôÑôÒ ÃåË! proc.memory.rssMB‚ 2841.859375 network.sent‚ 9.824242e+06 disk.md0.out‚ 98.37890625* proc.memory.percent‚17.345333099365234 proc.cpu.threads‚17( disk./.usagePercent‚76.3986999680681 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚9981.4150390625 disk.dm-0.in‚ 4.3359375" disk.dm-1.out‚5593.56787109375& disk.nvme0c0n1.out‚34211.408203125$ memory_percent‚96.99492454528809 cpu‚26.1057189220635 network.recv‚ 1.019836e+06 disk.nvme0n1.in‚ 4.3359375$ disk.nvme0n1.out‚34211.408203125 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625% disk./.usageGB‚1172.9110412597656 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚9981.4150390625# disk.dm-0.out‚38387.87353515625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375' proc.memory.availableMB‚ 492.3515625´î_J„‚(:’ ôÑôÒÞÒ¡Ì gpu.0.gpu‚92.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚262.98' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.66000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 r•¶2…- _wandbruntime‚3714  _runtime‚3714¦ãõÎ=†È{ @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7validation_average_jet_accuracy‚0.34643587470054626 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.28972333669662476 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6158989667892456 _runtime‚3714.476475711 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4765642285346985 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1511327624320984 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03542919456958771 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7828625440597534 +jet/accuracy_2_of_2‚0.15534646809101105 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6667564511299133 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.656020998954773 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9229432940483093 ,jet/accuracy_4_of_4‚0.018498167395591736 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 epoch‚8 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6480818390846252 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt0SRqqt2FBt/FRt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9967073798179626 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1698184311389923 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02294904552400112 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_of_4‚0.7683060169219971 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9971106648445129 particle/accuracy_5_of_5‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6661491990089417 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.993835985660553 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03002304770052433 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10000000149011612 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988153576850891 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9983673095703125 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6423611044883728 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9073967933654785 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12955404818058014 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10000000149011612 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978741407394409 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20958834886550903 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9943682551383972 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12604260444641113 +jet/accuracy_3_of_3‚0.08019131422042847 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6540684700012207 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9582472443580627 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6010134816169739 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6872086524963379 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01756897196173668 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9910383820533752 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7307424545288086 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9078947305679321 % _timestamp‚1.7844902379009917e+09 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2403426170349121 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8991227149963379 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3819444179534912 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.163534015417099 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14173978567123413 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18433260917663574 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_2‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20572210848331451 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9876282215118408 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7640160322189331 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7490673661231995 +jet/accuracy_3_of_4‚0.09701317548751831 particle/accuracy_2_of_4‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7265053391456604 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9876542687416077 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5403646230697632 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9960548877716064 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8974359035491943 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11936096847057343 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9876542687416077 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6893383860588074 particle/accuracy_1_of_4‚1 jet/accuracy_2_of_5‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9375 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16785551607608795 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9983673095703125 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_3‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36746588349342346 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12967956066131592 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5804545283317566 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1724773645401001 *jet/accuracy_1_of_4‚0.9758503437042236 particle/accuracy_3_of_4‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6496148109436035 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6629372239112854 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9914957880973816 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6163619756698608 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.46746671199798584 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8366763591766357 *jet/accuracy_1_of_1‚0.2997294068336487 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3207637369632721 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8366763591766357 particle/accuracy_1_of_2‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04211345314979553 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.011898341588675976 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.949999988079071 jet/accuracy_5_of_5‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18342962861061096 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7328016757965088 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9951896667480469 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 6 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.151791051030159 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8196581602096558 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1360027939081192 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9962481260299683 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8600000143051147 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.875 particle/accuracy_4_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3888888657093048 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.627364456653595 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.029690096154808998 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7247584462165833 (particle/accuracy‚0.8489194512367249 +particle/sensitivity‚0.6295944452285767 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4919331967830658 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.878180980682373 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.756247878074646 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1071428582072258 trainer/global_step‚7208 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8945615887641907 /particle/accuracy_4_of_4‚0.9354028105735779 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1635265052318573 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8600000143051147 *jet/accuracy_1_of_3‚0.9163920283317566 particle/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6949790120124817 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6300308108329773 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6151686906814575 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.021568726748228073 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 )loss/val_total_loss‚8.360035982406536 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2486703246831894 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.042306192219257355 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.42424246668815613 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45799145102500916 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8947368264198303 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1484275758266449 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6674946546554565 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16772018373012543 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 'particle/f_score‚0.6415064930915833 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9876436591148376 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4546703100204468 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1071428582072258 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 particle/accuracy_2_of_3‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36184388399124146 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.617412805557251 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9921368360519409 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15468749403953552 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt0SRqqt2FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7047659754753113 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6560108661651611 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5738823413848877 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6684442758560181 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11363097280263901 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9955961108207703 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01756897196173668 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9820767641067505 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5188345909118652 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7815026640892029 particle/accuracy_1_of_1‚1 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9772727489471436 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14723947644233704 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6744506359100342 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6163619756698608 +particle/specificity‚0.9088932871818542 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9464285969734192 particle/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16553683578968048 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7397933602333069 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03506626933813095 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4749999940395355 _step‚584 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980992078781128 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6533044576644897 *jet/accuracy_2_of_3‚0.5504871606826782 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9772727489471436 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1428496390581131 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7048275470733643 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1698184311389923 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.030412718653678894 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7620567083358765 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7654567956924438 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6266627311706543 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8942307829856873 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_5‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18448962271213531verfã‡Í{ 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6629372239112854 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9914957880973816 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6163619756698608 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.46746671199798584 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8366763591766357 *jet/accuracy_1_of_1‚0.2997294068336487 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3207637369632721 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8366763591766357 particle/accuracy_1_of_2‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04211345314979553 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.011898341588675976 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.949999988079071 jet/accuracy_5_of_5‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18342962861061096 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7328016757965088 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9951896667480469 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 6 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.151791051030159 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8196581602096558 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1360027939081192 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9962481260299683 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8600000143051147 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.875 particle/accuracy_4_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3888888657093048 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.627364456653595 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.029690096154808998 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7247584462165833 (particle/accuracy‚0.8489194512367249 +particle/sensitivity‚0.6295944452285767 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4919331967830658 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.878180980682373 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.756247878074646 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1071428582072258 trainer/global_step‚7208 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8945615887641907 /particle/accuracy_4_of_4‚0.9354028105735779 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1635265052318573 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8600000143051147 *jet/accuracy_1_of_3‚0.9163920283317566 particle/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6949790120124817 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6300308108329773 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6151686906814575 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.021568726748228073 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 )loss/val_total_loss‚8.360035982406536 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2486703246831894 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.042306192219257355 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.42424246668815613 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45799145102500916 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8947368264198303 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1484275758266449 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6674946546554565 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16772018373012543 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 'particle/f_score‚0.6415064930915833 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9876436591148376 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4546703100204468 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1071428582072258 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 particle/accuracy_2_of_3‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36184388399124146 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.617412805557251 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9921368360519409 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15468749403953552 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt0SRqqt2FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7047659754753113 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6560108661651611 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5738823413848877 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6684442758560181 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11363097280263901 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9955961108207703 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01756897196173668 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9820767641067505 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5188345909118652 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7815026640892029 particle/accuracy_1_of_1‚1 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9772727489471436 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14723947644233704 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6744506359100342 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6163619756698608 +particle/specificity‚0.9088932871818542 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9464285969734192 particle/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16553683578968048 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7397933602333069 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03506626933813095 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4749999940395355 _step‚584 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980992078781128 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6533044576644897 *jet/accuracy_2_of_3‚0.5504871606826782 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9772727489471436 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1428496390581131 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7048275470733643 9"äCPNðPurity/1FRt0SRqqt0FBt/event_purity‚0.1698184311389923 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.030412718653678894 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7620567083358765 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7654567956924438 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6266627311706543 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8942307829856873 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_5‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18448962271213531 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7validation_average_jet_accuracy‚0.34643587470054626 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.28972333669662476 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6158989667892456 _runtime‚3714.476475711 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4765642285346985 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1511327624320984 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03542919456958771 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7828625440597534 +jet/accuracy_2_of_2‚0.15534646809101105 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6667564511299133 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.656020998954773 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9229432940483093 ,jet/accuracy_4_of_4‚0.018498167395591736 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 epoch‚8 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6480818390846252 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt0SRqqt2FBt/FRt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9967073798179626 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1698184311389923 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02294904552400112 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_of_4‚0.7683060169219971 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9971106648445129 particle/accuracy_5_of_5‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6661491990089417 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.993835985660553 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03002304770052433 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10000000149011612 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988153576850891 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9983673095703125 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6423611044883728 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9073967933654785 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12955404818058014 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10000000149011612 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978741407394409 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20958834886550903 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9943682551383972 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12604260444641113 +jet/accuracy_3_of_3‚0.08019131422042847 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6540684700012207 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9582472443580627 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6010134816169739 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6872086524963379 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01756897196173668 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9910383820533752 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7307424545288086 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9078947305679321 % _timestamp‚1.7844902379009917e+09 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2403426170349121 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8991227149963379 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3819444179534912 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.163534015417099 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14173978567123413 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18433260917663574 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_2‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20572210848331451 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9876282215118408 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7640160322189331 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7490673661231995 +jet/accuracy_3_of_4‚0.09701317548751831 particle/accuracy_2_of_4‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7265053391456604 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9876542687416077 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5403646230697632 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9960548877716064 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8974359035491943 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11936096847057343 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9876542687416077 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6893383860588074 particle/accuracy_1_of_4‚1 jet/accuracy_2_of_5‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9375 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16785551607608795 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9983673095703125 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_3‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36746588349342346 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12967956066131592 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5804545283317566 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1724773645401001 *jet/accuracy_1_of_4‚0.9758503437042236 particle/accuracy_3_of_4‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6496148109436035 ("Purity/0FRt0SRqqt0FBt/event_purity‚1Èk @Øiˆ‚(:Þ ƒÒôÒ…üŽÌ network.sent‚ 9.840813e+06) disk./.usagePercent‚76.43124981513535 disk.md0.in‚0 disk.md0.out‚ 98.98046875" disk.dm-0.out‚38670.5751953125 disk.dm-1.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚34434.28271484375 network.recv‚ 1.023176e+06% disk./.usageGB‚1173.4107627868652 disk.dm-0.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚34434.28271484375& disk.nvme1c1n1.out‚10288.345703125# proc.memory.rssMB‚ 2844.33203125* proc.memory.percent‚17.360424995422363$ disk.nvme1n1.out‚10288.345703125" disk.dm-1.out‚5839.39599609375 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0% proc.memory.availableMB‚ 740.09375 proc.cpu.threads‚17 disk.nvme1n1.in‚0$ memory_percent‚95.48282623291016 cpu‚26.114529112326395z Ÿ‰‚(:” ƒÒôÒÝ„ÆÌ gpu.0.gpu‚53.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚157.942' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚52.647333333333336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷ù8!mŠ‚(:â ’ÒôÒèÓË disk.nvme0c0n1.in‚ 4.3359375* proc.memory.percent‚17.360424995422363 cpu‚26.112877444844514 network.sent‚ 9.884122e+06 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚10538.1748046875 disk.nvme1n1.in‚0# proc.memory.rssMB‚ 2844.33203125 proc.cpu.threads‚17 network.recv‚ 1.026783e+06& disk.nvme0n1.out‚35231.15087890625% disk.nvme1n1.out‚10538.1748046875 disk.md0.out‚ 99.23046875 disk.dm-1.in‚0" disk.dm-1.out‚6085.16552734375 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625$ disk./.usageGB‚1173.667854309082 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375( disk.nvme0c0n1.out‚35231.15087890625$ memory_percent‚95.48285007476807( proc.memory.availableMB‚ 740.08984375) disk./.usagePercent‚76.44799572115825 disk.nvme0n1.in‚ 4.3359375" disk.dm-0.out‚39470.9541015625,i½Ož‹‚(:“ ’ÒôÒ¶â™Ì gpu.0.gpu‚60.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚252.843' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.28099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YDj2Œ- _wandbruntime‚3736  _runtime‚3736U§Ÿ©ÇÁ % _timestamp‚1.7844902597358773e+09 _runtime‚3736.311364961 _step‚585 & lr-AdamW/pg1‚0.0001989867174262428 & lr-AdamW/pg2‚0.0001989867174262428 trainer/global_step‚7249†/{}ÌŽÆ % _timestamp‚1.7844902597358773e+09 _runtime‚3736.311364961 _step‚585 & lr-AdamW/pg1‚0.0001989867174262428 & lr-AdamW/pg2‚0.0001989867174262428 trainer/global_step‚7249É-3 2- _wandbruntime‚3736  _runtime‚3736½É­é'ãO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56563958 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7181620369408 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚549610601475120 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚556565765529188 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129905447 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚441223289792000 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚549610618514480 _runtime‚3736.311528131 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚555433229755904 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚555433229755904 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚556565408566506 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6967672195072 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6967331408896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57724327 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497472 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚562400919469568 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚555433247274496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57724620 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚441223265367552 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 129895302 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73331344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73331344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19207687 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 _step‚586 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6967672195072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73341487 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19207978 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚448404885736960 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56563960 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚562400561164800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚555433247274496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6967331408896 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7181751442432 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56563958 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56563960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38516642 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38516640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚562400561164800 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚448405041234432 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6955147014708 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚7249RÓ›î'‘èO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56563958 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56563960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38516642 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38516640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚562400561164800 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚448405041234432 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6955147014708 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚7249 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56563958 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7181620369408 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚549610601475120 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚556565765529188 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129905447 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚441223289792000 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚549610618514480 _runtime‚3736.311528131 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚555433229755904 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚555433229755904 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚556565408566506 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6967672195072 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6967331408896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57724327 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497472 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚562400919469568 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚555433247274496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57724620 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚441223265367552 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 129895302 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73331344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73331344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19207687 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 _step‚586 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6967672195072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73341487 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19207978 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚448404885736960 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56563960 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚562400561164800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚555433247274496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6967331408896 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7181751442432 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ÊÒrÆ2’- _wandbruntime‚3736  _runtime‚3736PA—¢“ìM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚562475682845184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚549684027557600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6968261320192 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56571462 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚556639762872222 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚549684044596960 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6968261320192 E=DeviceStatsMonitor.on_train_batch_e eP"nd/inactive_split.all.peak‚834 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 129912679 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6955735314622 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 $ _timestamp‚1.784490260110125e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚562476041157120 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57732145 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚448464723596288 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚555507421524992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚555507439043584 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73341219 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 trainer/global_step‚7249 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129922838 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129912679 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57731867 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73341219 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚562476041157120 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7182706670592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38521723 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚556640119835016 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73351376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚441282166124544 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56571462 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7182581896192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 _step‚587 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56571460 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6968602113536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38521721 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚441282141700096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 _runtime‚3736.685221781 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19210146 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129922838 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520|Â7­÷&”ñM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6955735314622 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 $ _timestamp‚1.784490260110125e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚562476041157120 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57732145 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚448464723596288 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚555507421524992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚555507439043584 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73341219 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 trainer/global_step‚7249 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129922838 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129912679 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57731867 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73341219 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚562476041157120 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7182706670592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38521723 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚556640119835016 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73351376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚441282166124544 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56571462 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7182581896192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 _step‚587 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56571460 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6968602113536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38521721 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚441282141700096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 _runtime‚3736.685221781 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19210146 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129922838 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚562475682845184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚549684027557600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6968261320192 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56571462 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚556639762872222 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚549684044596960 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6968261320192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 129912679 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720Ë)ÚÜ2•- _wandbruntime‚3736  _runtime‚3736¿1þ –† 2loss/SRqqt2/assignment_loss‚0.6668368943403016 0loss/FBt2/detection_loss‚0.47316855492277277 epoch‚9 3loss/FBt1/assignment_loss‚0.0010541099362229179 /loss/FRt1/detection_loss‚1.3268762454682705 /loss/FRt2/detection_loss‚0.5250879809933725 1loss/SRqqt1/detection_loss‚0.6070019776157682 0loss/FBt1/detection_loss‚0.48678759890149387 % _timestamp‚1.7844902601114056e+09 _runtime‚3736.686129192 _step‚588 %loss/total_loss‚8.095249760485835 2loss/SRqqt1/assignment_loss‚0.5588434967614746 2loss/FBt2/assignment_loss‚0.020084087641391888 1loss/SRqqt2/detection_loss‚0.9003569669131736 trainer/global_step‚7249 0loss/FRt1/assignment_loss‚0.7325239980922026 0loss/FRt2/assignment_loss‚1.7966278488993896 ó±Á—‹ 3loss/FBt1/assignment_loss‚0.0010541099362229179 /loss/FRt1/detection_loss‚1.3268762454682705 /loss/FRt2/detection_loss‚0.5250879809933725 1loss/SRqqt1/detection_loss‚0.6070019776157682 0loss/FBt1/detection_loss‚0.48678759890149387 % _timestamp‚1.7844902601114056e+09 _runtime‚3736.686129192 _step‚588 %loss/total_loss‚8.095249760485835 2loss/SRqqt1/assignment_loss‚0.5588434967614746 2loss/FBt2/assignment_loss‚0.020084087641391888 1loss/SRqqt2/detection_loss‚0.9003569669131736 trainer/global_step‚7249 0loss/FRt1/assignment_loss‚0.7325239980922026 0loss/FRt2/assignment_loss‚1.7966278488993896 2loss/SRqqt2/assignment_loss‚0.6668368943403016 0loss/FBt2/detection_loss‚0.47316855492277277 epoch‚9Ìuoe¼ž˜‚(:“ ¡ÒôÒ¨Ü×Ë gpu.0.gpu‚64.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚122.999' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.99966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O_-`™‚(:Õ ¡ÒôÒÔÖ¡Ì network.recv‚ 1.03003e+06$ disk./.usageGB‚1173.667854309082 disk.nvme1n1.in‚0& proc.memory.availableMB‚ 740.046875 cpu‚26.111422267799334 disk.dm-0.in‚ 4.3359375 network.sent‚ 9.902921e+06) disk./.usagePercent‚76.44799572115825 disk.nvme0n1.in‚ 4.3359375% disk.nvme0n1.out‚35241.7177734375$ disk.nvme1n1.out‚10540.955078125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375' disk.nvme0c0n1.out‚35241.7177734375$ memory_percent‚95.48311233520508# proc.memory.rssMB‚ 2844.33984375* proc.memory.percent‚17.360472679138184 proc.cpu.threads‚17# disk.dm-0.out‚39483.67822265625& disk.nvme1c1n1.out‚10540.955078125 disk.md0.in‚0 disk.md0.out‚99.5" disk.dm-1.out‚6085.20458984375œ6áè2š- _wandbruntime‚3761  _runtime‚3761ÔS½zț ' lr-AdamW/pg1‚0.00019877578055427878 ' lr-AdamW/pg2‚0.00019877578055427878 trainer/global_step‚7299 % _timestamp‚1.7844902852099426e+09 _runtime‚ 3761.78492982 _step‚589ðh´~ÍœÇ % _timestamp‚1.7844902852099426e+09 _runtime‚ 3761.78492982 _step‚589 ' lr-AdamW/pg1‚0.00019877578055427878 ' lr-AdamW/pg2‚0.00019877578055427878 trainer/global_step‚7299Íg÷‹ò2- _wandbruntime‚3761  _runtime‚3761 ‹¦eé'žãO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 _runtime‚3761.789457408 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7013859230720 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56939060 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73826946 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚566156677441024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚7299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚566156677441024 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56939058 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 _step‚590 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7013859230720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58104625 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚553281905599120 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73837089 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 130776149 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚444167081994752 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7014200016896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19334219 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19333935 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260~Ëo/ŸèO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56939060 =7DeviceStatsMonitor.on_train_batch_start/num_devi>šM¿&ce_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73826946 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚566156677441024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚7299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚566156677441024 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56939058 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 _step‚590 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7013859230720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58104625 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚553281905599120 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73837089 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 130776149 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚444167081994752 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7014200016896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19334219 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19333935 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 _runtime‚3761.789457408 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7013859230720 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130783381 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚566232157433344 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚553355348720960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56946562 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56946560 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73836821 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7230523578368 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚553355331681600 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130783381 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚559217009979392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73846978 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73846978 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚559217027497984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚559217027497984 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73836821 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130793540 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56946560 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚560357867024024 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 $ _timestamp‚1.784490285588206e+09 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58112213 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7002518303064 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚566231799121408 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38775773è@Þ÷&¢ñM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚566231799121408 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38775773 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19336730 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58112503 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7014789142016 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7015129935360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚444225982751744 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚444225958327296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚7299 _step‚591 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚560357510061230 _runtime‚3762.163318239 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7014789142016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19336442 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7230652547072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56946562 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚451456481905664 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130783381 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚566232157433344 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚553355348720960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56946562 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56946560 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73836821 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7230523578368 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚553355331681600 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130783381 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚559217009979392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73846978 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73846978 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚559217027497984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚559217027497984 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73836821 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130793540 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56946560 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚560357867024024 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 $ _timestamp‚1.784490285588206e+09 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58112213 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7002518303064 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0Ï' O‚2£- _wandbruntime‚3762  _runtime‚37627I2¤ % _timestamp‚1.7844902855892224e+09 .loss/FRt2/detection_loss‚0.620865109924081 1loss/SRqqt1/detection_loss‚0.6558982748085909 %loss/total_loss‚8.642976322935787 0loss/FRt2/assignment_loss‚1.8960501331893551 2loss/SRqqt1/assignment_loss‚0.5961043491768476 2loss/SRqqt2/assignment_loss‚0.5030630229712187 0loss/SRqqt2/detection_loss‚0.964664053992072 1loss/FBt2/assignment_loss‚0.16219678542276111 /loss/FRt1/detection_loss‚1.2342376689960206 /loss/FBt2/detection_loss‚0.7145310445972111 trainer/global_step‚7299 _runtime‚3762.164063581 _step‚592 /loss/FRt1/assignment_loss‚0.807621141981824 3loss/FBt1/assignment_loss‚0.0029440530934212727 0loss/FBt1/detection_loss‚0.48480068478238536 epoch‚9äoq ¥† 0loss/FRt2/assignment_loss‚1.8960501331893551 2loss/SRqqt1/assignment_loss‚0.5961043491768476 2loss/SRqqt2/assignment_loss‚0.5030630229712187 0loss/SRqqt2/detection_loss‚0.964664053992072 1loss/FBt2/assignment_loss‚0.16219678542276111 /loss/FRt1/detection_loss‚1.2342376689960206 /loss/FBt2/detection_loss‚0.7145310445972111 trainer/global_step‚7299 _runtime‚3762.164063581 _step‚592 /loss/FRt1/assignment_loss‚0.807621141981824 3loss/FBt1/assignment_loss‚0.0029440530934212727 0loss/FBt1/detection_loss‚0.48480068478238536 epoch‚9 % _timestamp‚1.7844902855892224e+09 .loss/FRt2/detection_loss‚0.620865109924081 1loss/SRqqt1/detection_loss‚0.6558982748085909 %loss/total_loss‚8.642976322935787Ðé¡éMo¦‚(:ä °ÒôÒÿƒÉË# proc.memory.rssMB‚ 2844.33984375 proc.cpu.threads‚17 network.sent‚ 9.951378e+06& disk.nvme1n1.out‚10562.56005859375$ memory_percent‚95.48342227935791 cpu‚26.109799225453592 disk.rbd0.in‚0 disk.md0.in‚0 disk.md0.out‚ 99.90234375 disk.dm-0.in‚ 4.3359375" disk.dm-0.out‚40289.5908203125 disk.dm-1.in‚0( disk.nvme0c0n1.out‚36030.67919921875 disk.nvme0n1.in‚ 4.3359375 network.recv‚ 1.034522e+06! disk.dm-1.out‚6085.5107421875 disk.nvme0c0n1.in‚ 4.3359375( disk.nvme1c1n1.out‚10562.56005859375( proc.memory.availableMB‚ 739.99609375* proc.memory.percent‚17.360472679138184 disk.rbd0.out‚ 0.4697265625) disk./.usagePercent‚76.61221269248189% disk./.usageGB‚1176.1889953613281 disk.nvme1n1.in‚0& disk.nvme0n1.out‚36030.67919921875 disk.nvme1c1n1.in‚05WBî3§‚(:” °ÒôÒöíÌ gpu.0.gpu‚57.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚191.371' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚63.790333333333336 gpu.0.smCloci'*mlk‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñì®l¨‚(:á ¿ÒôÒâ€ÊË network.recv‚ 1.038661e+06% disk./.usageGB‚1176.6673774719238 disk.dm-0.in‚ 4.3359375 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375( disk.nvme1c1n1.out‚10607.75634765625 disk.nvme1n1.in‚0 network.sent‚ 1.0015288e+07 disk.dm-1.in‚0% disk.nvme0n1.out‚37361.8427734375# proc.memory.rssMB‚ 2844.33984375* proc.memory.percent‚17.360472679138184 cpu‚26.108640153281826 proc.cpu.threads‚17# disk.dm-0.out‚41668.12060546875! disk.dm-1.out‚6085.7138671875 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0 disk.md0.out‚100.4375( proc.memory.availableMB‚ 739.94140625) disk./.usagePercent‚76.64337257592729 disk.rbd0.in‚0' disk.nvme0c0n1.out‚37361.8427734375& disk.nvme1n1.out‚10607.75634765625 disk.md0.in‚0$ memory_percent‚95.48375606536865රyž©‚(:“ ¿ÒôÒà‘ˆÌ gpu.0.gpu‚91.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚120.982' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.32733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TF<¨2ª- _wandbruntime‚3786  _runtime‚3786ŽãÒ°É«Ã _step‚593 ' lr-AdamW/pg1‚0.00019856336449118007 ' lr-AdamW/pg2‚0.00019856336449118007 trainer/global_step‚7349 % _timestamp‚1.7844903104088533e+09 _runtime‚3786.983843372cÈEòÎ¬È ' lr-AdamW/pg1‚0.00019856336449118007 ' lr-AdamW/pg2‚0.00019856336449118007 trainer/global_step‚7349 % _timestamp‚1.7844903104088533e+09 _runtime‚3786.983843372 _step‚593ѯ‡V²2­- _wandbruntime‚3786  _runtime‚3786ëεêè'®âO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7277632139776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131646851 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚454388555186176 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7060727838720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19460698 trainer/global_step‚7349 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚569913152022016 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57314160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚562852424183296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7047693221402 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58485438 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7060387052544 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚454388399688704 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57314160 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74332691 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57314158 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚556953226762480 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74332691 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚562852424183296 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74322548 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚564001259907204 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57314158 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7060387052544 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚569912793717248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 _runtime‚3786.987455208 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19460982 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131636706 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39024740 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7277501066752 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 $ _timestamp‚1.784490310409861e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚562852406664704 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚447110923046400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚447110898621952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚556953209723120 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚562852406664704 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚569912793717248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚569913152022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74322548 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚564000902944522 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7060727838720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58485724 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39024742 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚569912793717248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 _runtime‚3786.987455208 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19460982 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131636706 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39024740 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7277501066752 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 $ _timestamp‚1.784490310409861e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚562852406664704 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚447110923046400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚447110898621952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚556953209723120 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚562852406664704 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚569912793717248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚569913152022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74322548 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚564000902944522 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0Ò€¤ÅF2°- _wandbruntime‚3787  _runtime‚3787Nß5ó&±íM BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7048961368072 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131654083 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7061316963840 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚569988273709568 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚454448388004864 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57321660 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚562926615952384 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131664242 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39029821 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚569987915397632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚562926598433792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7061316963840 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚562926598433792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚447169774954496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74332423 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58493293 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19463470 trainer/global_step‚7349 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58493023 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39029823 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57321660 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚564075614213032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131654083 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 _runtime‚3787.361622019 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57321662 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7061657757184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚447169799378944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7278588625920 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚557026635805600 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57321662 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚569988273709568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚454448238806016 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚557026652844960 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74342580 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7278463851520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74332423 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434O‡"¤A²òM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 L>DeviceStatsMonitor.on_train_batch_end/allocat¹ûn·%ion.all.allocated‚ 131664242 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39029821 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚569987915397632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚562926598433792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7061316963840 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚562926598433792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚447169774954496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74332423 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58493293 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19463470 trainer/global_step‚7349 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58493023 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39029823 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57321660 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚564075614213032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131654083 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 _runtime‚3787.361622019 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57321662 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7061657757184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚447169799378944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7278588625920 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚557026635805600 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57321662 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚569988273709568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚454448238806016 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚557026652844960 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74342580 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7278463851520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74332423 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7048961368072 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131654083 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7061316963840 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚569988273709568 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚454448388004864 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57321660 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚562926615952384 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ÓôÓ\ 2³- _wandbruntime‚3787  _runtime‚3787“L’ ´ 2loss/FBt2/assignment_loss‚0.004002678325752009 .loss/FRt1/detection_loss‚1.421530997288685 0loss/FBt1/detection_loss‚0.39966918648256416 trainer/global_step‚7349 _runtime‚3787.362680491 2loss/SRqqt1/assignment_loss‚0.5535934048490532 epoch‚9 /loss/FBt2/detection_loss‚0.6521748401882017 0loss/FRt1/assignment_loss‚0.8804885896809427 0loss/FRt2/assignment_loss‚1.9040179764241547 2loss/SRqqt2/assignment_loss‚0.7447001522194351 .loss/FRt2/detection_loss‚0.515246599283473 1loss/SRqqt2/detection_loss‚1.0150566479619245 % _timestamp‚1.7844903107879214e+09 1loss/FBt1/assignment_loss‚0.09485457514858021 1loss/SRqqt1/detection_loss‚0.6306959411041622 _step‚596 %loss/total_loss‚8.816031588956928Ï“c µ† 2loss/FBt2/assignment_loss‚0.004002678325752009 .loss/FRt1/detection_loss‚1.421530997288685 0loss/FBt1/detection_loss‚0.39966918648256416 trainer/global_step‚7349 _runtime‚3787.362680491 2loss/SRqqt1/assignment_loss‚0.5535934048490532 epoch‚9 /loss/FBt2/detection_loss‚0.6521748401882017 0loss/FRt1/assignment_loss‚0.8804885896809427 0loss/FRt2/assignment_loss‚1.9040179764241547 2loss/SRqqt2/assignment_loss‚0.7447001522194351 .loss/FRt2/detection_loss‚0.515246599283473 1loss/SRqqt2/detection_loss‚1.0150566479619245 % _timestamp‚1.7844903107879214e+09 1loss/FBt1/assignment_loss‚0.09485457514858021 1loss/SRqqt1/detection_loss‚0.6306959411041622 _step‚596 %loss/total_loss‚8.816031588956928Ô¸ír¶‚(:ç ÎÒôÒÈŸ¼Ë* proc.memory.percent‚17.360472679138184& disk.nvme0n1.out‚37362.86083984375 network.recv‚ 1.042093e+06 disk.dm-0.in‚ 4.3359375 disk.nvme1n1.in‚0& disk.nvme1n1.out‚10608.61279296875 disk.dm-1.in‚0 disk.md0.out‚ 100.92578125# memory_percent‚95.4840898513794 cpu‚26.107098035124338 proc.cpu.threads‚17 network.sent‚ 1.0019055e+07) disk./.usagePercent‚76.64337804235556# disk.dm-0.out‚41668.74951171875" disk.dm-1.out‚6085.90771484375 disk.rbd0.out‚ 0.4697265625( disk.nvme0c0n1.out‚37362.86083984375 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚10608.61279296875% disk./.usageGB‚1176.6674613952637 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 disk.md0.in‚0( proc.memory.availableMB‚ 739.88671875# proc.memory.rssMB‚ 2844.33984375"'7ž·‚(:“ ÎÒôÒË™¬Ì gpu.0.gpu‚65.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚174.229' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚58.07633333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F¼çgb¸‚(:× ÝÒôÒ…†íË disk.md0.out‚ 101.25390625 disk.dm-0.in‚ 4.3359375 cpu‚26.10556103902977 proc.cpu.threads‚17 network.recv‚ 1.045869e+06$ disk./.usageGB‚1176.680908203125" disk.dm-1.out‚6085.98583984375 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 disk.nvme1n1.in‚0' proc.memory.availableMB‚ 739.3671875# proc.memory.rssMB‚ 2844.33984375 network.sent‚ 1.0082473e+07) disk./.usagePercent‚76.64425391324703 disk.rbd0.out‚ 0.4697265625# disk.nvme0c0n1.out‚ 37992.953125$ disk.nvme1n1.out‚10884.552734375# disk.dm-0.out‚42573.97998046875$ memory_percent‚95.48726081848145 disk.dm-1.in‚0 disk.rbd0.in‚0! disk.nvme0n1.out‚ 37992.953125 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚10884.552734375* proc.memory.percent‚17.360472679138184 disk.md0.in‚0ÒÓŸž¹‚(:“ ÝÒôÒË·Ì gpu.0.gpu‚81.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚252.294' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.09800000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µÛ/Å2º- _wandbruntime‚3812  _runtime‚3812½¯6Å»¿ % _timestamp‚1.7844903359583201e+09 _runtime‚3812.534407223 _step‚597 % lr-AdamW/pg1‚0.000198349472639221 % lr-AdamW/pg2‚0.000198349472639221 trainer/global_step‚7399´ÆUÊ¼Ä % _timestamp‚1.7844903359583201e+09 _runtime‚3812.534407223 _step‚597 % lr-AdamW/pg1‚0.000198349472639221 % lr-AdamW/pg2‚0.000198349472639221 trainer/global_step‚7399ÕNEß2½- _wandbruntime‚3812  _runtime‚3812‹w=è'¾âO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7107255660544 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74828293 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39278790 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7106914874368 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚457380369809920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚457380210118144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 135267328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7106914874368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19587108 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57689260 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74828293 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573668909993472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19586831 trainer/global_step‚7399 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚573669268298240 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57689260 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚573668909993472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚567719007096212 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57689258 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚573669268298240 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚560624513847120 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58865900 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39278792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 $ _timestamp‚1.784490335959026e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74818150 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7325630136320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57689258 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 132507408 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7107255660544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚567718650133530 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 132517553 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚3812.534493064 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7094136286410 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚566561995119104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚560624530886480 _step‚598 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159691776 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚450054715249152 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573668909993472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19586831 trainer/global_step‚7399 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚573669268298240 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57689260 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚573668909993472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚567719007096212 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57689258 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚573669268298240 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚560624513847120 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58865900 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39278792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 $ _timestamp‚1.784490335959026e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74818150 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7325630136320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57689258 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 132507408 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7107255660544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚567718650133530 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 132517553 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚3812.534493064 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7094136286410 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚566561995119104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚560624530886480 _step‚598 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159691776 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚450054715249152 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/act]ìa6ive_bytes.large_pool.current‚17518592 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7106914874368 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚457380369809920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚457380210118144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 135267328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7106914874368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19587108 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57689260 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74828293 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132534944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚566636186888192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7108185579008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚560697939929600 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57696762 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74838182 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚450113591581696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚560697956968960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19589568 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132534944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚573744031673856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7107844785664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161781760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7326584730112 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚567793361402040 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚457440200736256 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74828025 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚573744389985792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚566636204406784 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39283871 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚566636186888192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57696760 $ _timestamp‚1.784490336332612e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 132524785 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚450113616006144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7107844785664 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 137357312 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74828025 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7095064509646 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7108185579008 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7326447372800 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58873441 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57696762 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚573744389985792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19589297 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚457440038954496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚7399 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58873168 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚567793004439246 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚566636204406784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39283873 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ¾{;÷&ÂñM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132534944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚573744031673856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7107844785664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161781760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7326584730112 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚567793361402040 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚457440200736256 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74828025 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚573744389985792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚566636204406784 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39283871 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚566636186888192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57696760 $ _timestamp‚1.784490336332612e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 132524785 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚450113616006144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7107844785664 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 137357312 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74828025 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7095064509646 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7108185579008 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7326447372800 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58873441 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57696762 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚573744389985792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19589297 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚457440038954496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚7399 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58873168 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚567793004439246 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚566636204406784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39283873 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 _runtime‚3812.907738118 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132534944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚566636186888192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7108185579008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚560697939929600 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57696762 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74838182 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚450113591581696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚560697956968960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19589568×I€M–2Ã- _wandbruntime‚3812  _runtime‚3812#ÑÛfÄ .loss/FRt2/detection_loss‚0.591487067898468 1loss/SRqqt1/detection_loss‚0.7227643394170288 trainer/global_step‚7399 /loss/FBt1/detection_loss‚0.5076349065317933 /loss/FRt1/detection_loss‚1.3797984780846038 1loss/SRqqt2/detection_loss‚0.8969120812591512 /loss/FBt2/detection_loss‚1.0743489187955857 $ _timestamp‚1.784490336333698e+09 _runtime‚3812.908392209 0loss/FRt1/assignment_loss‚0.9117020908492564 3loss/SRqqt2/assignment_loss‚0.47048891238058743 3loss/FBt1/assignment_loss‚0.0017162675633820147 1loss/FBt2/assignment_loss‚0.03141880260539637 $loss/total_loss‚9.04535665742474 epoch‚9 _step‚600 0loss/FRt2/assignment_loss‚1.8864082944583134 2loss/SRqqt1/assignment_loss‚0.5706764975811748 Ù¾ ņ _step‚600 0loss/FRt2/assignment_loss‚1.8864082944583134 2loss/SRqqt1/assignment_loss‚0.5706764975811748 .loss/FRt2/detection_loss‚0.591487067898468 1loss/SRqqt1/detection_loss‚0.7227643394170288 trainer/global_step‚7399 /loss/FBt1/detection_loss‚0.5076349065317933 /loss/FRt1/detection_loss‚1.3797984780846038 1loss/SRqqt2/detection_loss‚0.8969120812591512 /loss/FBt2/detection_loss‚1.0743489187955857 $ _timestamp‚1.784490336333698e+09 _runtime‚3812.908392209 0loss/FRt1/assignment_loss‚0.9117020908492564 3loss/SRqqt2/assignment_loss‚0.47048891238058743 3loss/FBt1/assignment_loss‚0.0017162675633820147 1loss/FBt2/assignment_loss‚0.03141880260539637 $loss/total_loss‚9.04535665742474 epoch‚9Ø\;G@lÆ‚(:á ìÒôÒòÝ¡Ì$ memory_percent‚95.48609256744385( proc.memory.availableMB‚ 739.55859375 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚10885.408203125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375' disk.nvme0c0n1.out‚38022.5380859375 disk.dm-1.in‚0" disk.dm-1.out‚6086.36181640625 network.sent‚ 1.0100169e+07) disk./.usagePercent‚76.64420496386671 disk.nvme1c1n1.in‚0 disk.md0.out‚ 101.6171875# disk.dm-0.out‚42603.25732421875 disk.rbd0.out‚ 0.4697265625& disk.nvme1c1n1.out‚10885.408203125 disk.dm-0.in‚ 4.3359375% disk.nvme0n1.out‚38022.5380859375# proc.memory.rssMB‚ 2844.33984375* proc.memory.percent‚17.360472679138184 cpu‚26.104165869877953 proc.cpu.threads‚17 network.recv‚ 1.049683e+06% disk./.usageGB‚1176.6801567077637 disk.md0.in‚0 disk.nvme0n1.in‚ 4.3359375ã¦ëžÇ‚(:“ ìÒôÒƒÛÌ gpu.0.gpu‚56.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚196.922' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.64066666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü°2È- _wandbruntime‚3838  _runtime‚3838Ñ@(IÇÉÁ _runtime‚3838.073857556 _step‚601 & lr-AdamW/pg1‚0.0001981341084243138 & lr-AdamW/pg2‚0.0001981341084243138 trainer/global_step‚7449 % _timestamp‚1.7844903614988568e+09}I¦ÌÊÆ _runtime‚3838.073857556 _step‚601 & lr-AdamW/pg1‚0.0001981341084243138 & lr-AdamW/pg2‚0.0001981341084243138 trainer/global_step‚7449 % _timestamp‚1.7844903614988568e+09وǘV2Ë- _wandbruntime‚3838  _runtime‚3838H‘_ÌâO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7153783482368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚571436754285220 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75313752 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚452998556300800 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚570271583573504 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 KBDeviceStatsMonitor.on_train_batch_start/allocation¬¬á .large_pool.peak‚1446 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39532840 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚460372125798400 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 _step‚602 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7153783482368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133388255 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚452998531876352 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚577425026269696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7153442696192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58064358 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75323895 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59245778 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7140919274740 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675gâÙí'ÍçO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39532840 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚460372125798400 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 _step‚602 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7153783482368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133388255 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚452998531876352 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚577425026269696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7153442696192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58064358 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75323895 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59245778 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7140919274740 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7153783482368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚571436754285220 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75313752 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚452998556300800 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚570271583573504 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935Ú^3‰2Î- _wandbruntime‚3838  _runtime‚3838—gužò&ÏìM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75333784 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59253336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58071860 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚564369261092960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7154713400832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7154713400832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58071860 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19715132 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39537921 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7154372607488 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39537923 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚564369244053600 _step‚603 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75323627 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚453057408208896 trainer/global_step‚7449 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚571511108591048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚570345792861184 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚577500147950080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 $ _timestamp‚1.784490361875577e+09 _runtime‚3838.450690049 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚570345792861184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75333784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚460431963118592 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚571510751628254 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133395487 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7374530485248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7141847498088 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59253053 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚460431807628288 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58071862 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133405646 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7141507574654 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7154372607488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7374399419392 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚577500506262016 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58071862 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2þÉ-ë$ÐñM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75323627 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19715413 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚570345775342592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133405646 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7141507574654 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7154372607488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7374399419392 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚577500506262016 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58071862 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75333784 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59253336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58071860 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚564369261092960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7154713400832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7154713400832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58071860 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19715132 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39537921 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7154372607488 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39537923 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚564369244053600 _step‚603 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75323627 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚453057408208896 trainer/global_step‚7449 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚571511108591048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚570345792861184 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚577500147950080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 $ _timestamp‚1.784490361875577e+09 _runtime‚3838.450690049 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚570345792861184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75333784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚460431963118592 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚571510751628254 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133395487 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7374530485248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7141847498088 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59253053 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚460431807628288 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚CV÷ 0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58071862 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚573981172027904 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚581181500850688 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7199970518016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚581181142545920 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59625117 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 134248812 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75819497 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7421451195904 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19838227 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59625398 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7187362339748 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 trainer/global_step‚7499 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚573981189546496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58439458 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7187022416426 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75809354 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚581181500850688 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7200311304192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 134258957 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚567967139134480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7844903870195434e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚573981189546496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7200311304192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚575154144511546 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7421584366080 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚573981172027904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚567967122095120 _step‚606 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58439460 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682†z$Úí'ÝçO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚573981172027904 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚581181500850688 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7199970518016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚581181142545920 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59625117 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 134248812 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75819497 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7421451195904 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19838227 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59625398 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7187362339748 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 trainer/global_step‚7499 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚573981189546496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58439458 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7187022416426 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75809354 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚581181500850688 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7200311304192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 134258957 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚567967139134480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7844903870195434e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚573981189546496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7200311304192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚575154144511546 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7421584366080 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚573981172027904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚567967122095120 _step‚606 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58439460 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0Þù³xD2Þ- _wandbruntime‚3863  _runtime‚3863ºaε1ßíM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚574055363796992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚574055363796992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚581256264226304 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75829386 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58446960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7187950639662 _step‚607 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58446962 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7200900429312 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚568040565216960 trainer/global_step‚7499 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59632911 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135260160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 134276348 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7201241222656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7200900429312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 134266189 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19840651 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚581256622538240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 134266189 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚581256622538240 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134276348 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 VIDeviceStatsMonitor.on_hI§Â train_batch_end/inactive_split.small_pool.allocated‚19840938 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7422540598272 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58446962 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75829386 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚463423789858816 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚456001224836096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚568040548177600 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 % _timestamp‚1.7844903873944871e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59632622 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7187950639662 _step‚607 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58446962 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7200900429312 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚568040565216960 trainer/global_step‚7499 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59632911 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135260160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 134276348 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7201241222656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7200900429312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 134266189 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19840651 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚581256622538240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 134266189 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚581256622538240 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134276348 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19840938 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7422540598272 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58446962 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75829386 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚463423789858816 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚456001224836096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚568040548177600 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 % _timestamp‚1.7844903873944871e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59632622 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58446960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚466355716393472 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 _step‚610 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76315099 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19964319 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚7549 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚577690778000896 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76315099 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135119514 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135129659 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76304956 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58814558 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚584937258822144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚577690760482304 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7246498339840 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 % _timestamp‚1.7844904122516046e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚458886189555200 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚584937617126912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40040940 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7233805404756 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚577690760482304 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚584937258822144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7246839126016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60005538 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚578871891700554 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7469526838272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76304956 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚458886165130752 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58814560 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚578872248663236 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚466355558798848 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚466355716393472 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 _step‚610 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76315099 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19964319 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚7549 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚577690778000896 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76315099 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135119514 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135129659 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76304956 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58814558 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚584937258822144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚577690760482304 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7246498339840 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 % _timestamp‚1.7844904122516046e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚458886189555200 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚584937617126912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40040940 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7233805404756 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚577690760482304 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚584937258822144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7246839126016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60005538 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚578871891700554 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7469526838272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76304956 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚458886165130752 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58814560 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚578872248663236 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚466355558798848 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260â!¹EÌ2î- _wandbruntime‚3889  _runtime‚3889 2†ó&ïíM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60012863 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76324988 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 _step‚611 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚585012738814464 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚466415383645696 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7247428251136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7470483734016 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58822062 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76324988 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135136891 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7234733628104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58822060 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚571711869340960 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40046021 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚571711852301600 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7234393704670 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 trainer/global_step‚7549 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚585012738814464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚466415549621760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60013149 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚458945065887744 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141551616 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135147050 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19967126 % _timestamp‚1.7844904126275213e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚577764952251392 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚585012380502528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135147050 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58822060 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135136891 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚577764969769984 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚577764952251392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7247428251136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58822062 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7247769044480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚578946246006270 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7247769044480 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚578946602969064 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675(!Yø&ðòM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76324988 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 _step‚611 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚585012738814464 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚466415383645696 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7247428251136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7470483734016 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58822062 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76324988 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135136891 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7234733628104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58822060 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚571711869340960 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40046021 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚571711852301600 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7234393704670 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 trainer/global_step‚7549 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚585012738814464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚466415549621760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60013149 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚458945065887744 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141551616 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135147050 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19967126 % _timestamp‚1.7844904126275213e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚577764952251392 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚585012380502528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135147050 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58822060 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135136891 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚577764969769984 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚577764952251392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7247428251136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58822062 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7247769044480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚578946246006270 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7247769044480 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚578946602969064 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60012863ã¹Éwï2ñ- _wandbruntime‚3889  _runtime‚3889÷·ßÕò‚ /loss/FRt2/detection_loss‚0.5670109124292129 1loss/SRqqt2/detection_loss‚0.8507505545082191 _runtime‚3889.203387218 0loss/FRt2/assignment_loss‚1.8854922483144871 $ _timestamp‚1.784490412628629e+09 _step‚612 0loss/FRt1/assignment_loss‚0.8962016682075374 2loss/SRqqt2/assignment_loss‚0.6154368020963438 2loss/FBt1/assignment_loss‚0.004697267084582571 /loss/FBt1/detection_loss‚0.6290791709828427 trainer/global_step‚7549 2loss/FBt2/assignment_loss‚0.001005095359640773 1loss/SRqqt1/detection_loss‚0.6631708815482831 /loss/FBt2/detection_loss‚1.1750549513380975 %loss/total_loss‚9.274535965962507 epoch‚9 2loss/SRqqt1/assignment_loss‚0.5397080910421825 /loss/FRt1/detection_loss‚1.4469283230510759ïQu€ ó‡ 1loss/SRqqt2/detection_loss‚0.8507505545082191 _runtime‚3889.203387218 0loss/FRt2/assignment_loss‚1.8854922483144871 $ _timestamp‚1.784490412628629e+09 _step‚612 0loss/FRt1/assignment_loss‚0.8962016682075374 2loss/SRqqt2/assignment_loss‚0.6154368020963438 2loss/FBt1/assignment_loss‚0.004697267084582571 /loss/FBt1/detection_loss‚0.6290791709828427 trainer/global_step‚7549 2loss/FBt2/assignment_loss‚0.001005095359640773 1loss/SRqqt1/detection_loss‚0.6631708815482831 /loss/FBt2/detection_loss‚1.1750549513380975 %loss/total_loss‚9.274535965962507 epoch‚9 2loss/SRqqt1/assignment_loss‚0.5397080910421825 /loss/FRt1/detection_loss‚1.4469283230510759 /loss/FRt2/detection_loss‚0.5670109124292129ä9?6dô‚(:Ù ·ÓôÒ ¶ëË disk.rbd0.out‚ 0.4697265625( disk.nvme0c0n1.out‚38037.24365234375 disk.nvme1c1n1.in‚0 disk.md0.in‚0# memory_percent‚95.4864501953125$ disk./.usageGB‚1176.681728363037" disk.dm-0.out‚42619.3134765625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 4.3359375 disk.nvme1n1.in‚0& disk.nvme1n1.out‚10890.32275390625# proc.memory.rssMB‚ 2844.33984375 disk.dm-0.in‚ 4.3359375 disk.dm-1.out‚6086.884765625 disk.nvme0c0n1.in‚ 4.3359375& disk.nvme0n1.out‚38037.24365234375* proc.memory.percent‚17.360472679138184 cpu‚26.09693799511927 proc.cpu.threads‚17 network.sent‚ 1.0295904e+07 network.recv‚ 1.068542e+06) disk./.usagePercent‚76.64430733515955( disk.nvme1c1n1.out‚10890.32275390625 disk.md0.out‚ 103.015625! proc.memory.availableMB‚739.5S§hžõ‚(:“ ·ÓôÒ…ߢÌ gpu.0.gpu‚96.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚232.139' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚77.37966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Mý@2ö- _wandbruntime‚3914  _runtime‚3914ã]îÆ÷À & lr-AdamW/pg2‚0.0001974792162144362 trainer/global_step‚7599 $ _timestamp‚1.784490437768246e+09 _runtime‚3914.343240515 _step‚613 & lr-AdamW/pg1‚0.0001974792162144362¼°»ªËøÅ trainer/global_step‚7599 $ _timestamp‚1.784490437768246e+09 _runtime‚3914.343240515 _step‚613 & lr-AdamW/pg1‚0.0001974792162144362 & lr-AdamW/pg2‚0.0001974792162144362åáápû2ù- _wandbruntime‚3914  _runtime‚3914}2$àúãO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7517354990592 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7293026161664 _step‚614 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76810701 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20090358 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚575309747382480 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7293026161664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60385348 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 % _timestamp‚1.7844904377692773e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59189660 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59189660 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20090649 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚469347490148864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚581400348936704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚581400348936704 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136000361 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚588693375098368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7279908546442 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135990216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚588693375098368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40294990 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚3914.344181127 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7293366947840 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚581400366455296 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚469347336748544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7517483966464 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76810701 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60385641 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 trainer/global_step‚7599 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0àyiaî'ûèO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7517354990592 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7293026161664 _step‚614 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76810701 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20090358 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚575309747382480 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7293026161664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60385348 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 % _timestamp‚1.7844904377692773e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59189660 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59189660 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20090649 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚469347490148864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚581400348936704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚581400348936704 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136000361 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚588693375098368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7279908546442 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135990216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚588693375098368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40294990 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚3914.344181127 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7293366947840 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚581400366455296 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚469347336748544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7517483966464 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76810701 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60385641 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 trainer/global_step‚7599 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0æ}xÛ$2ü- _wandbruntime‚3914  _runtime‚39140¯˜ó&ýíM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7518315221504 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚575383156425600 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136017752 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59197162 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚469407326705152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚588768496778752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚582663993195278 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136007593 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 _step‚615 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7293956072960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59197162 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7294296866304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60392952 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 _runtime‚3914.719477982 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚581474540705792 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20092881 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136017752 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40300073 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚575383173464960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76810433 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136007593 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76820590 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76810433 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚581474540705792 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 % _timestamp‚1.7844904381443434e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40300071 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7294296866304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20093171 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚461888882514944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60393244 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7280836769678 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59197160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚588768855090688 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚581474558224384 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7293956072960 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚461888858090496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7518444190208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76820590 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚582664350158072 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 þôïþòM YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7281176693112 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7294296866304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60392952 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 _runtime‚3914.719477982 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚834 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚581474540705792 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20092881 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136017752 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40300073 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚575383173464960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76810433 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136007593 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76820590 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76810433 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚581474540705792 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 % _timestamp‚1.7844904381443434e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40300071 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7294296866304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20093171 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚461888882514944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60393244 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7280836769678 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59197160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚588768855090688 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚581474558224384 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7293956072960 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚461888858090496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7518444190208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76820590 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚582664350158072 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.curren¯å®ø t‚ 340793344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7518315221504 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚575383156425600 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136017752 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59197162 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚469407326705152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚588768496778752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚582663993195278 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136007593 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 _step‚615 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7293956072960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59197162 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7339553983488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7565340482048 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77296160 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122684416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 trainer/global_step‚7649 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59564760 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚592449491374592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚585109937391104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136871063 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚464773822809600 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7326351611450 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 % _timestamp‚1.7844904633031244e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚585109937391104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60765452 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40549042 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚472339138867200 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚578981034467120 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147108864 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚592449491374592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7339553983488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚586307743041252 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚592449849679360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚472339285976064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚585109954909696 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40549040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 136860918 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59564758 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7565463166464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚464773798385152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7339894769664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260Îþׯî'‹èO JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚592449491374592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7339553983488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚586307743041252 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚592449849679360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚490 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚472339285976064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚585109954909696 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40549040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 136860918 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59564758 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7565463166464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚464773798385152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7339894769664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59564758 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7339553983488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7565340482048 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77296160 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122684416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 trainer/global_step‚7649 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59564760 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚834 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚592449491374592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚585109937391104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136871063 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚464773822809600 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7326351611450 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 % _timestamp‚1.7844904633031244e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚585109937391104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60765452 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40549042 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚472339138867200 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚578981034467120 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147108864 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚586381740384286 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7327619758120 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136888454 trainer/global_step‚7649 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7340824688128 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77306035 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20218612 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚472398968576000 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77316192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚592524971366912 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 _step‚619 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60772733 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚296 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136878295 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136888454 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚585184129160192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚585184146678784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59572260 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7340483894784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40554123 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚586382097347080 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2600`_Ç÷&ŽñM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60773031 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚586382097347080 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚585184129160192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133163008 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚586381740384286 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7327619758120 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136888454 trainer/global_step‚7649 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7340824688128 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77306035 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20218612 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚472398968576000 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77316192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚592524971366912 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 _step‚619 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60772733 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚296 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136878295 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136888454 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚585184129160192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚585184146678784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59572260 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7340483894784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40554123 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚490 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚596205607650816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7386081805312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚467717615012352 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚590025490230260 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚582652355630480 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 % _timestamp‚1.7844904888576632e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚475331089912832 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚588819525845504 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77801905 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7372794676458 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124781568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40803092 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚590025133267578 _step‚622 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77801905 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7386081805312 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 137741765 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59939858 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7373134599780 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 137731620 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7613325694464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚582652338591120 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61145262 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7386422591488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20342456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 _runtime‚3965.432681583 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59939860 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚596205607650816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59939860 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚475330940706816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚588819525845504 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77791762 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7386422591488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚588819543364096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77791762 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7386422591488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚588819543364096 ZLDeviceStatsMonitor.on_train_batchnd“îQ"_start/inactive_split_bytes.small_pool.peak‚ 179347968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚596205607650816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7386081805312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚467717615012352 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚590025490230260 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚582652355630480 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 % _timestamp‚1.7844904888576632e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚475331089912832 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚588819525845504 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77801905 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7372794676458 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124781568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40803092 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚590025133267578 _step‚622 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77801905 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7386081805312 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 137741765 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59939858 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7373134599780 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 137731620 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7613325694464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚582652338591120 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61145262 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7386422591488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20342456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 _runtime‚3965.432681583 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59939860 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚596205607650816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59939860 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚475330940706816îbJÎÈ2œ- _wandbruntime‚3965  _runtime‚3965ŸTÝò&ìM ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚582725764673600 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 137748997 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚588893717614592 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚596280729331200 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 137759156 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59947362 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77801637 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7387011716608 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚590099844536088 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 _step‚623 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 _runtime‚ 3965.80761199 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7614408221696 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7387352509952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20344664 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚582725781712960 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚588893735133184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚475390770597888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7614279252992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚7699 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7374062823128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚590099487573294 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 137748997 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚588893717614592 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 % _timestamp‚1.7844904892324665e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40808171 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7387011716608 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7387352509952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40808173 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59947360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77811794 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 137759156 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77811794 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61153121 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚596281087643136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61152835 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0€|¯Ã÷&žñM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚590099487573294 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 137748997 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚588893717614592 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 % _timestamp‚1.7844904892324665e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40808171 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7387011716608 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7387352509952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40808173 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59947360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77811794 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 137759156 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77811794 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61153121 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚596281087643136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61152835 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚582725764673600 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 137748997 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚588893717614592 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚596280729331200 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 137759156 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59947362 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77801637 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7387011716608 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚590099844536088 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 _step‚623 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 _runtime‚ 3965.80761199 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7614408221696 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7387352509952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20344664 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚582725781712960 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚588893735133184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚475390770597888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7614279252992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚7699 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7374062823128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675ï=W‚2Ÿ- _wandbruntime‚3965  _runtime‚3965r6¿Ú  … 0loss/FRt2/assignment_loss‚1.8138121719897913 3loss/FBt1/assignment_loss‚0.0007357902013015478 4loss/FBt2/assignment_loss‚0.00042516511427379553 /loss/FRt2/detection_loss‚0.5741534612198299 1loss/SRqqt1/detection_loss‚0.5893911880705304 trainer/global_step‚7699 /loss/FBt2/detection_loss‚0.8977358402761941 %loss/total_loss‚8.498258200386605 _runtime‚3965.808405391 0loss/FRt1/assignment_loss‚0.6859502462530325 2loss/SRqqt1/assignment_loss‚0.5315952556265198 2loss/SRqqt2/assignment_loss‚0.5903984815238182 /loss/FRt1/detection_loss‚1.4012900831936141 % _timestamp‚1.7844904892335613e+09 0loss/SRqqt2/detection_loss‚1.005299474630091 /loss/FBt1/detection_loss‚0.4074710422876076 epoch‚9 _step‚624Ýs‡‰¡Š 0loss/SRqqt2/detection_loss‚1.005299474630091 /loss/FBt1/detection_loss‚0.4074710422876076 epoch‚9 _step‚624 0loss/FRt2/assignment_loss‚1.8138121719897913 3loss/FBt1/assignment_loss‚0.0007357902013015478 4loss/FBt2/assignment_loss‚0.00042516511427379553 /loss/FRt2/detection_loss‚0.5741534612198299 1loss/SRqqt1/detection_loss‚0.5893911880705304 trainer/global_step‚7699 /loss/FBt2/detection_loss‚0.8977358402761941 %loss/total_loss‚8.498258200386605 _runtime‚3965.808405391 0loss/FRt1/assignment_loss‚0.6859502462530325 2loss/SRqqt1/assignment_loss‚0.5315952556265198 2loss/SRqqt2/assignment_loss‚0.5903984815238182 /loss/FRt1/detection_loss‚1.4012900831936141 % _timestamp‚1.7844904892335613e+09ðÛ*ü6h¢‚(:Ý ‚ÔôÒج¹Ë disk.dm-1.in‚0 disk.rbd0.in‚0% disk./.usageGB‚1176.7438697814941% disk.nvme0c0n1.out‚38108.02734375 disk.nvme1n1.in‚0 disk.rbd0.out‚ 0.4697265625 cpu‚26.0901068086035 disk.nvme0n1.in‚ 4.3359375( disk.nvme1c1n1.out‚10894.01904296875& disk.nvme1n1.out‚10894.01904296875 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375! disk.dm-1.out‚6087.5361328125$ memory_percent‚95.48726081848145' proc.memory.availableMB‚ 739.3671875 disk.md0.out‚ 104.8984375# disk.dm-0.out‚42689.11474609375# proc.memory.rssMB‚ 2844.33984375* proc.memory.percent‚17.360472679138184 proc.cpu.threads‚17 network.sent‚ 1.0479304e+07 network.recv‚ 1.087311e+06) disk./.usagePercent‚76.64835497681122 disk.nvme0c0n1.in‚ 4.3359375# disk.nvme0n1.out‚38108.02734375 disk.nvme1c1n1.in‚0ØÝÀŽ“£‚(:ˆ ‚ÔôÒ—Ì gpu.0.gpu‚97.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚245.259' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚81.753 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kv˜dk¤‚(:à ‘ÔôÒ½ñÉË# proc.memory.rssMB‚ 2844.33984375* proc.memory.percent‚17.360472679138184 proc.cpu.threads‚17 network.sent‚ 1.0543553e+07 network.recv‚ 1.091176e+06) disk./.usagePercent‚76.64835497681122 disk.md0.out‚ 105.171875$ memory_percent‚96.99134826660156 cpu‚26.088768453156888% disk./.usageGB‚1176.7438697814941" disk.dm-0.out‚42692.6591796875( disk.nvme1c1n1.out‚10894.91552734375 disk.md0.in‚0 disk.nvme0c0n1.in‚ 4.3359375( disk.nvme0c0n1.out‚38111.31591796875 disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚38111.31591796875 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0! disk.dm-1.out‚6087.5634765625 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0& disk.nvme1n1.out‚10894.91552734375$ proc.memory.availableMB‚492.9375V;æA”¥‚(:“ ‘ÔôÒì¬ÿË gpu.0.gpu‚71.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 149¨<$ 57740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚234.265' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚78.08833333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬ª4N2¦- _wandbruntime‚3990  _runtime‚3990šn—Áɧà trainer/global_step‚7749 % _timestamp‚1.7844905144096816e+09 _runtime‚3990.985715179 _step‚625 ' lr-AdamW/pg1‚0.00019681119832848994 ' lr-AdamW/pg2‚0.00019681119832848994wšÓãÎ¨È ' lr-AdamW/pg1‚0.00019681119832848994 ' lr-AdamW/pg2‚0.00019681119832848994 trainer/global_step‚7749 % _timestamp‚1.7844905144096816e+09 _runtime‚3990.985715179 _step‚625ñI¹õ2©- _wandbruntime‚3990  _runtime‚3990!±!é'ªãO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7661441732096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚586323659754480 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7419577664788 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚592529114299904 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚307 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20468937 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚478322725521408 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚470661456064000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚470661431639552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7661293881856 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78287364 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 _step‚626 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60314958 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78297507 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚599961723927040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7432950413312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚599962082231808 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚478322897796096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 138612467 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚592529131818496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7432950413312 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78287364 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41057142 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚593743237419268 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138602322 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61526079 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚599961723927040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 _runtime‚3990.985779579 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚586323642715120 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚592529131818496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41057140 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60314958 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78297507 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7432609627136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7432609627136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚309 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 172274688 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 % _timestamp‚1.7844905144106662e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374GC0î'«èO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 147850240 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚593742880456586 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 _runtime‚3990.985779579 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚586323642715120 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚592529131818496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41057140 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60314958 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78297507 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7432609627136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7432609627136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚309 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 172274688 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 % _timestamp‚1.7844905144106662e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7661441732096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚586323659754480 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7419577664788 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚592529114299904 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚307 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20468937 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚478322725521408 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚470661456064000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚470661431639552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7661293881856 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78287364 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 _step‚626 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60314958 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78297507 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚599961723927040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7432950413312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚599962082231808 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚478322897796096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 138612467 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚592529131818496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7432950413312 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78287364 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41057142 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚593743237419268 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138602322 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61526079 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚599961723927040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚592603323587584 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚600036845607424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚586397085836960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚470720332396544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚586397068797600 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚593817591725096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60322462 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚470720307972096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41062223 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61533348 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7420505888136 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138629858 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7433539538432 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚627 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60322460 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 _runtime‚3991.361579576 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚478382721566720 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135260160 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 trainer/global_step‚7749 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7662253910016 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138619699 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087%•‚ñ¹®òM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60322462 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7433539538432 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7433880331776 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚600037203919360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚592603306068992 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61533641 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7433880331776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚600036845607424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHÌ‚ ¶?!DeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚586397085836960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚470720332396544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚586397068797600 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚593817591725096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60322462 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚470720307972096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41062223 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61533348 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7420505888136 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138629858 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7433539538432 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚627 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60322460 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 _runtime‚3991.361579576 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚478382721566720 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135260160 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 trainer/global_step‚7749 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7662253910016 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138619699 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚592603323587584 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚473605272691200 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 % _timestamp‚1.7844905396019123e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60690058 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139461632 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41311190 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60690060 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7709401765888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚589994946839120 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78782966 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚603717840203264 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20595136 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚589994963878480 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7479478235136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41311192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7465680806474 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚481314674457088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚597460984608276 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61906328 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚596238702754304 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60690058 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚603717840203264 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20594848 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163886080 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚596238720272896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7479137448960 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7479137448960 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78782966 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚603718198508032 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7466020729796 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 _runtime‚4016.177490907 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837|jŒñî'¹èO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚596238720272896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7479137448960 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7479137448960 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78782966 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚603718198508032 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7466020729796 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 _runtime‚4016.177490907 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚630 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60690060 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚596238702754304 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚603718198508032 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61906038 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78793109 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚481314510571008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 trainer/global_step‚7799 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚473605248266752 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚473605272691200 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 % _timestamp‚1.7844905396019123e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60690058 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139461632 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41311190 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60690060 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7709401765888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚589994946839120 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78782966 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚603717840203264 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20595136 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚589994963878480 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7479478235136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41311192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7465680806474 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚481314674457088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚597460984608276 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61906328 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚596238702754304 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60690058 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚603717840203264 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20594848 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163886080 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675ö†Ï í2º- _wandbruntime‚4016  _runtime‚4016Å%¿ »íM YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.onõ&Aœá%_train_batch_end/segment.small_pool.peak‚260 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60697562 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7710222106624 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 _step‚631 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61913910 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20597637 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚473664149023744 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41316271 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 139500560 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61913618 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚596312912041984 _runtime‚4016.551868825 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 139490401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20597347 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7466948953144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60697560 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚596312894523392 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60697562 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78802998 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚590068372921600 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 trainer/global_step‚7799 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚473664124599296 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚597534981951310 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7480067360256 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78792841 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7480408153600 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚596312894523392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚481374346705920 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚603793320195584 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7480067360256 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78802998 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚596312912041984 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139500560 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0pé{ø&¼òM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41316273 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78792841 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7480408153600 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 139490401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61913618 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚596312912041984 _runtime‚4016.551868825 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 139490401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20597347 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7466948953144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60697560 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚596312894523392 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60697562 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78802998 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚590068372921600 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 trainer/global_step‚7799 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚473664124599296 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚597534981951310 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7480067360256 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78792841 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7480408153600 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚596312894523392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚481374346705920 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚603793320195584 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7480067360256 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78802998 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚596312912041984 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139500560 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60697562 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7710222106624 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 _step‚631 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61913910 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20597637 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚473664149023744 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41316271 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 139500560 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 EDÛ¿Š 0loss/FRt1/assignment_loss‚0.7812370443151064 2loss/SRqqt1/assignment_loss‚0.5096158168375192 1loss/SRqqt2/detection_loss‚1.0194407579089915 /loss/FBt2/detection_loss‚0.5819924314320087 epoch‚9 trainer/global_step‚7799 2loss/FBt1/assignment_loss‚0.003925113307272207 4loss/FBt2/assignment_loss‚0.00036271187158916975 _step‚632 /loss/FRt2/assignment_loss‚1.927151226051665 /loss/FRt1/detection_loss‚1.3505583989381156 /loss/FRt2/detection_loss‚0.5785612066286154 % _timestamp‚1.7844905399778562e+09 2loss/SRqqt2/assignment_loss‚0.6774757886636087 1loss/SRqqt1/detection_loss‚0.5800495565442058 0loss/FBt1/detection_loss‚0.38769721852925915 %loss/total_loss‚8.398067271027957 _runtime‚4016.553015427øôûgÀ‚(:Ü ¯ÔôÒçˆÌ$ disk./.usageGB‚1176.743881225586% disk.nvme1c1n1.out‚10899.82421875( proc.memory.availableMB‚ 492.83203125# proc.memory.rssMB‚ 2844.33984375* proc.memory.percent‚17.360472679138184 proc.cpu.threads‚17 disk.dm-0.in‚ 4.3359375' disk.nvme0c0n1.out‚38115.8447265625 disk.nvme0n1.in‚ 4.3359375 disk.nvme1n1.in‚0 network.sent‚ 1.0622893e+07) disk./.usagePercent‚76.64835572223325 disk.md0.in‚0# disk.dm-0.out‚42697.11669921875 disk.rbd0.in‚0% disk.nvme0n1.out‚38115.8447265625# disk.nvme1n1.out‚10899.82421875 cpu‚26.08611843732884 network.recv‚ 1.099218e+06 disk.md0.out‚ 105.890625 disk.dm-1.in‚0" disk.dm-1.out‚6090.97119140625 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0$ memory_percent‚96.991991996765144ß0œÁ‚(:‘ ¯ÔôÒ¶ê»Ì gpu.0.gpu‚58.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚266.3' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.76666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€¹L´n‚(:ã ¾ÔôÒåÛôË network.sent‚ 1.0673288e+07% disk./.usageGB‚1176.7438621520996% disk.nvme0n1.out‚38116.8427734375( disk.nvme1c1n1.out‚10900.59912109375 disk.md0.out‚ 106.3046875 disk.dm-1.in‚0 cpu‚26.084690090157313 proc.cpu.threads‚17) disk./.usagePercent‚76.64835447986319 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375& disk.nvme1n1.out‚10900.59912109375 network.recv‚ 1.103789e+06 disk.md0.in‚0# disk.dm-0.out‚42697.79248046875 disk.dm-1.out‚6091.181640625$ memory_percent‚96.99199199676514# proc.memory.rssMB‚ 2844.33984375' disk.nvme0c0n1.out‚38116.8427734375 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.dm-0.in‚ 4.3359375( proc.memory.availableMB‚ 492.83203125* proc.memory.percent‚17.360472679138184÷Ê挞Â(:“ ¾ÔôÒ‹Þ¨Ì gpu.0.gpu‚91.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚256.406' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.46866666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ\+2Ä- _wandbruntime‚4041  _runtime‚4041”4ÎÉÅà % _timestamp‚1.7844905650854359e+09 _runtime‚4041.660448309 _step‚633 ' lr-AdamW/pg1‚0.00019635860850503254 ' lr-AdamW/pg2‚0.00019635860850503254 trainer/global_step‚7849¦5ŸŸÎÆÈ % _timestamp‚1.7844905650854359e+09 _runtime‚4041.660448309 _step‚633 ' lr-AdamW/pg1‚0.00019635860850503254 ' lr-AdamW/pg2‚0.00019635860850503254 trainer/global_step‚7849ù¥bÅa2Ç- _wandbruntime‚4041  _runtime‚40417@38n ÈãO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚599948291208704 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61065158 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79278568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7512463794804 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚593666268002480 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚607474314784256 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚476549064893952 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7525665270784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79288711 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62286007 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7526006056960 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7512123871482 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7757217739264 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚601178731797284 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7849 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61065160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚607473956479488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7525665270784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20721062 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚484306282633216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7844905650864937e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41565242 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚295 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61065160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚601178374834602 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79288711 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7526006056960 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140353871 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.alj]t<{located‚607474314784256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62286304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7757357200896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139461632 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚599948291208704 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 _step‚634 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79288711 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7526006056960 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140353871 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚607474314784256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62286304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7757357200896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139461632 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚599948291208704 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 _step‚634 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7525665270784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79288711 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62286007 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7526006056960 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7512123871482 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7757217739264 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚601178731797284 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7849 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61065160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚607473956479488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7525665270784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20721062 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚484306282633216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7844905650864937e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41565242 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚295 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61065160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚601178374834602 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79288443 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7758308902912 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚601252729140318 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7526935975424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61072662 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚607549078159872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20723269 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844905654637442e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7526595182080 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140371262 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61072660 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140361103 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62293590 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7513052094718 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79298600 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7526595182080 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚601253086103112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚484366274553856 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7513392018152 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79288443 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79298600 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61072662 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62293879 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚600022500496384 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7526935975424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚593739694084960 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚476607941226496 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41570323 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚484366119063552 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚600022482977792 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 _runtime‚ 4042.03887215 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140371262 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61072660 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41570321 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚476607965650944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7849Ë_ù÷&ÌñM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140361103 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62293590 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7513052094718 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79298600 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7526595182080 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚601253086103112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚484366274553856 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7513392018152 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79288443 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79298600 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61072662 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62293879 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚600022500496384 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7526935975424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚593739694084960 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚476607941226496 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41570323 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚484366119063552 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚600022482977792 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 _runtime‚ 4042.03887215 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140371262 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61072660 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41570321 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚476607965650944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7849 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚600022500496384 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20723556 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79288443 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7758308902912 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚601252729140318 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7526935975424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61072662 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚607549078159872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20723269 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844905654637442e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7526595182080 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140371262 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61072660 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0û¬KC32Í- _wandbruntime‚4042  _runtime‚4042Ã@“΄ 0loss/FRt2/assignment_loss‚1.8563220665658682 2loss/SRqqt1/assignment_loss‚0.6316524127287061 /loss/FRt1/detection_loss‚1.4675277319638969 epoch‚9 trainer/global_step‚7849 3loss/FBt2/assignment_loss‚0.0004547740091766173 /loss/FRt2/detection_loss‚0.5823801847535061 1loss/SRqqt1/detection_loss‚0.6613863847563096 1loss/SRqqt2/detection_loss‚0.7785600710382558 %loss/total_loss‚8.972055707566373 2loss/SRqqt2/assignment_loss‚0.6075609043176916 /loss/FBt2/detection_loss‚1.209689533Þnéñ3379507 % _timestamp‚1.7844905654648206e+09 _runtime‚4042.043410878 0loss/FRt1/assignment_loss‚0.7776870349226477 2loss/FBt1/assignment_loss‚0.013545447247588908 /loss/FBt1/detection_loss‚0.3852891619247732 _step‚636ý¥Å™Ï‰ 3loss/FBt2/assignment_loss‚0.0004547740091766173 /loss/FRt2/detection_loss‚0.5823801847535061 1loss/SRqqt1/detection_loss‚0.6613863847563096 1loss/SRqqt2/detection_loss‚0.7785600710382558 %loss/total_loss‚8.972055707566373 2loss/SRqqt2/assignment_loss‚0.6075609043176916 /loss/FBt2/detection_loss‚1.2096895333379507 % _timestamp‚1.7844905654648206e+09 _runtime‚4042.043410878 0loss/FRt1/assignment_loss‚0.7776870349226477 2loss/FBt1/assignment_loss‚0.013545447247588908 /loss/FBt1/detection_loss‚0.3852891619247732 _step‚636 0loss/FRt2/assignment_loss‚1.8563220665658682 2loss/SRqqt1/assignment_loss‚0.6316524127287061 /loss/FRt1/detection_loss‚1.4675277319638969 epoch‚9 trainer/global_step‚7849üyP@kЂ(:à ÍÔôÒõ®×Ë network.sent‚ 1.067621e+07 network.recv‚ 1.106865e+06) disk./.usagePercent‚76.64785181693739% disk./.usageGB‚1176.7361450195312 disk.md0.in‚0 disk.md0.out‚ 106.6171875 disk.nvme0c0n1.in‚ 4.3359375& disk.nvme0n1.out‚38118.07666015625 disk.nvme1c1n1.in‚0" disk.dm-0.out‚42698.9814453125 disk.dm-1.out‚6091.318359375$ memory_percent‚96.99239730834961& proc.memory.availableMB‚ 492.765625# proc.memory.rssMB‚ 2844.33984375* proc.memory.percent‚17.360472679138184 disk.nvme1n1.in‚0 disk.rbd0.in‚0 cpu‚26.08339631952993 proc.cpu.threads‚17( disk.nvme1c1n1.out‚10901.37451171875& disk.nvme1n1.out‚10901.37451171875 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625( disk.nvme0c0n1.out‚38118.07666015625 disk.nvme0n1.in‚ 4.3359375g-%žÑ‚(:“ ÍÔôÒ—ͽÌ gpu.0.gpu‚82.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚263.441' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.81366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t×NlÒ‚(:á ÜÔôÒ‰€×Ë network.sent‚ 1.0740566e+07 disk.dm-0.in‚ 4.3359375& disk.nvme0n1.out‚38119.16357421875 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0& proc.memory.availableMB‚ 492.765625 disk.md0.in‚0 disk.md0.out‚ 107.01171875" disk.dm-0.out‚42699.7314453125 disk.dm-1.in‚0 disk.rbd0.in‚0( disk.nvme1c1n1.out‚10902.09130859375$ memory_percent‚96.99239730834961 network.recv‚ 1.11128e+06$ disk./.usageGB‚1176.506435394287 disk.nvme0c0n1.in‚ 4.3359375( disk.nvme0c0n1.out‚38119.16357421875& disk.nvme1n1.out‚10902.09130859375 cpu‚26.08204771375414 proc.cpu.threads‚17) disk./.usagePercent‚76.63288945737092! disk.dm-1.out‚6091.5263671875 disk.rbd0.out‚ 0.4697265625 disk.nvme0n1.in‚ 4.3359375# proc.memory.rssMB‚ 2844.33984375* proc.memory.percent‚17.360472679138184§@ˆÓ‚(:’ ÜÔôÒƒÉ‡Ì gpu.0.gpu‚89.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚254.22' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.74000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³’Q2Ô- _wandbruntime‚4067  _runtime‚4067uc ¸ÇÕÁ _step‚637 & lr-AdamW/pg1‚0.0001961301510624079 & lr-AdamW/pg2‚0.0001961301510624079 trainer/global_step‚7899 % _timestamp‚1.7844905905980048e+09 _runtime‚4067.173006758,ÈD[ÌÖÆ _runtime‚4067.173006758 _step‚637 & lr-AdamW/pg1‚0.0001961301510624079 & lr-AdamW/pg2‚0.0001961301510624079 trainer/global_step‚7899 % _timestamp‚1.7844905905980048e+09ýÇåÈB2×- _wandbruntime‚4067  _runtime‚4067AëyYç'ØáO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 % _timestamp‚1.7844905905991473e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79774170 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚597337572126480 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7805350074880 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 _runtime‚ 4067.1741687 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚603657879663104 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚603657897181696 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 trainer/global_step‚7899 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚487298102620160 _step‚638 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41819290 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7558906859812 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61440260 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61440258 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7572193092608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79784313 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7572193092608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41819292 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7558566936490 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚479492881521152 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20847010 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚611230431060480 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚603657897181696 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61440258 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61440260 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚611230072755712 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7805350074880 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 _runtime‚ 4067.1741687 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚603657879663104 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚603657897181696 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 trainer/global_step‚7899 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚487298102620160 _step‚638 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41819290 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7558906859812 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61440260 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61440258 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7572193092608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79784313 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7572193092608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41819292 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7558566936490 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚479492881521152 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20847010 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚611230431060480 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚603657897181696 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61440258 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61440260 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚611230072755712 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 % _timestamp‚1.7844905905991473e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79774170 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚597337572126480 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0þE„<2Ú- _wandbruntime‚4067  _runtime‚4067ëô&ÛíM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7573463797248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62673648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 147101696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚597410998208960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚611305552748032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7573123003904 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚597410981169600 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20849277 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7559495159726 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚603732088950784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚611305194436096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 _step‚639 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41824373 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41824371 trainer/global_step‚7899 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7806306241536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚604970476329326 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141241964 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122677248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7559835083160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚603732071432192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚487357941417984 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7806183564288 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61447762 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141231805 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79794202 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚611305552748032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141231805 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141241964 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79784045 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 % _timestamp‚1.7844905909746406e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚479551782278144 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚603732088950784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61447762 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62673922 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79794202 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceZ­?~cStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79784045 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61447760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7573123003904 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚479551757853696 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61447760 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚487358088519680 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7559835083160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚603732071432192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚487357941417984 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7806183564288 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61447762 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141231805 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79794202 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚611305552748032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141231805 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141241964 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79784045 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 % _timestamp‚1.7844905909746406e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚479551782278144 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚603732088950784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61447762 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62673922 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79794202 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79784045 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61447760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7573123003904 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚479551757853696 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61447760 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚487358088519680 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62673648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 147101696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚597410998208960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚611305552748032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7573123003904 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚597410981169600 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20849277 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7559495159726 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚603732088950784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚611305194436096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 _step‚639 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41824373 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41824371 trainer/global_step‚7899 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7806306241536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚604970476329326 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141241964 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675ÿ¾Ðî&2Ý- _wandbruntime‚4067  _runtime‚4067O',Þ€ %loss/total_loss‚9.016440189851238 trainer/global_step‚7899 % _timestamp‚1.7844905909756918e+09 /loss/FRt2/assignment_loss‚1.750940171815868 epoch‚9 1loss/SRqqt1/detection_loss‚0.6631430576287578 .loss/FBt1/detection_loss‚0.539951608566424 0loss/FRt1/assignment_loss‚0.8436432154022129 2loss/SRqqt1/assignment_loss‚0.5712903183080713 2loss/FBt1/assignment_loss‚0.022982950848059694 3loss/FBt2/assignment_loss‚0.0006771586417114252 1loss/SRqqt2/detection_loss‚0.9279229383902637 /loss/FBt2/detection_loss‚0.9774763166372266 _runtime‚4067.550804386 _step‚640 2loss/SRqqt2/assignment_loss‚0.5796776578118605 .loss/FRt1/detection_loss‚1.501212838539156 .loss/FRt2/detection_loss‚0.637521957261626½×õè ß… .loss/FRt1/detection_loss‚1.501212838539156 .loss/FRt2/detection_loss‚0.637521957261626 %loss/total_loss‚9.016440189851238 trainer/global_step‚7899 % _timestamp‚1.7844905909756918e+09 /loss/FRt2/assignment_loss‚1.750940171815868 epoch‚9 1loss/SRqqt1/detection_loss‚0.6631430576287578 .loss/FBt1/detection_loss‚0.539951608566424 0loss/FRt1/assignment_loss‚0.8436432154022129 2loss/SRqqt1/assignment_loss‚0.5712903183080713 2loss/FBt1/assignment_loss‚0.022982950848059694 3loss/FBt2/assignment_loss‚0.0006771586417114252 1loss/SRqqt2/detection_loss‚0.9279229383902637 /loss/FBt2/detection_loss‚0.9774763166372266 _runtime‚4067.550804386 _step‚640 2loss/SRqqt2/assignment_loss‚0.5796776578118605€¥DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61815360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7853299841024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚607367485636096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7619061700608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚608614226175300 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142085130 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7619061700608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42073342 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚614986189031936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚614986189031936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80279915 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7618720914432 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80269772 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚482436722572800 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚601008876250480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚608613869212618 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7605349924820 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61815360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61815358 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20972521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20972800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7605010001498 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚490290022413824 trainer/global_step‚7949 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80279915 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63046142 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7853172962304¡œi¶çæO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61815360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61815358 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20972521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20972800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7605010001498 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚490290022413824 trainer/global_step‚7949 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80279915 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63046142 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7853172962304 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61815358 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7618720914432 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7844906151424277e+09 _runtime‚ 4091.7172622 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142095275 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61815360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7853299841024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚607367485636096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7619061700608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚608614226175300 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142085130 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7619061700608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42073342 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚614986189031936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚614986189031936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80279915 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7618720914432 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80269772 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚482436722572800 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚601008876250480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚608613869212618 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7605349924820 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0‚•œ™2è- _wandbruntime‚4092  _runtime‚40923bQ]ó&éíM E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20975026 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚490349709543424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122677248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 _step‚643 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80279647 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80289804 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚615061669024256 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7619991619072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7854257739776 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63053447 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚615061310712320 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚482495574480896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7606278148168 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚601082285293600 % _timestamp‚1.7844906155002866e+09 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142102507 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61822862 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61822860 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7619650825728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚607441677405184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80289804 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20975304 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚608688223518334 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 _runtime‚4092.075435912 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 142102507 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61822862 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚608688580481128 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61822860 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142112666 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42078423 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142112666 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42078421 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚615061669024256 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7619991619072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7854257739776 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63053447 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚615061310712320 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚482495574480896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7606278148168 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚601082285293600 % _timestamp‚1.7844906155002866e+09 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142102507 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61822862 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61822860 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7619650825728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚607441677405184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80289804 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20975304 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚608688223518334 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 _runtime‚4092.075435912 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 142102507 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61822862 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚608688580481128 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61822860 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142112666 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42078423 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142112666 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42078421 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80279647 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80289804 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520ƒáëQ2ë- _wandbruntime‚4092  _runtime‚4092­I÷¥ì€ _step‚644 2loss/SRqqt2/assignment_loss‚0.6735574757049401 .loss/FRt1/detection_loss‚1.417189801896089 /loss/FBt1/detection_loss‚0.7195759023821284 0loss/FRt2/assignment_loss‚1.9526847106179381 2loss/FBt1/assignment_loss‚0.002864214725284647 /loss/FBt2/detection_loss‚0.4941499611651081 _runtime‚4092.076554194 2loss/SRqqt1/assignment_loss‚0.5874362362407589 -loss/FRt2/detection_loss‚0.50749664824108 %loss/total_loss‚8.872526798913455 epoch‚9 trainer/global_step‚7949 % _timestamp‚1.7844906155019002e+09 0loss/FRt1/assignment_loss‚0.7891276044056827 2loss/FBt2/assignment_loss‚0.011692255665572812 1loss/SRqqt1/detection_loss‚0.5534293738286467 1loss/SRqqt2/detection_loss‚1.1633226140402257²‘p_ í… 0loss/FRt1/assignment_loss‚0.7891276044056827 2loss/FBt2/assignment_loss‚0.011692255665572812 1loss/SRqqt1/detection_loss‚0.5534293738286467 1loss/SRqqt2/detection_loss‚1.1633226140402257 _step‚644 2loss/SRqqt2/assignment_loss‚0.6735574757049401 .loss/FRt1/detection_loss‚1.417189801896089 /loss/FBt1/detection_loss‚0.7195759023821284 0loss/FRt2/assignment_loss‚1.9526847106179381 2loss/FBt1/assignment_loss‚0.002864214725284647 /loss/FBt2/detection_loss‚0.4941499611651081 _runtime‚4092.076554194 2loss/SRqqt1/assignment_loss‚0.5874362362407589 -loss/FRt2/detection_loss‚0.50749664824108 %loss/total_loss‚8.872526798913455 epoch‚9 trainer/global_step‚7949 % _timestamp‚1.7844906155019002e+09„4Æ›žî‚(:“ úÔôÒô›ªË gpu.0.gpu‚91.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚218.948' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚72.98266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽÚãdjï‚(:ß úÔôÒÊ…ÒÍ' proc.memory.availableMB‚ 492.6953125 proc.cpu.threads‚17' disk./.usagePercent‚76.632848956107% disk./.usageGB‚1176.5058135986328 disk.dm-1.in‚0" disk.dm-1.out‚6091.74755859375 disk.nvme0c0n1.in‚ 4.3359375& disk.nvme0c0n1.out‚38131.115234375 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚10903.79443359375 network.recv‚ 1.118828e+06 disk.rbd0.in‚0 disk.dm-0.in‚ 4.3359375! disk.dm-0.out‚42711.021484375$ memory_percent‚96.99282646179199# proc.memory.rssMB‚ 2844.33984375* proc.memory.percent‚17.360472679138184 cpu‚26.081848447053318 disk.nvme0n1.in‚ 4.3359375$ disk.nvme0n1.out‚38131.115234375 disk.md0.in‚0 network.sent‚ 1.0807133e+07 disk.rbd0.out‚ 0.4697265625( disk.nvme1c1n1.out‚10903.79443359375 disk.nvme1n1.in‚0 disk.md0.out‚ 108.02734375§Pûdoð‚(:ä ‰ÕôÒÌüË network.sent‚ 1.0870356e+07 network.recv‚ 1.123126e+06& disk.nvme0n1.out‚38143.68701171875' disk.nvme1c1n1.out‚10904.8662109375% disk.nvme1n1.out‚10904.8662109375 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375$ memory_percent‚96.99294567108154% disk./.usageGB‚1176.5057945251465 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1n1.in‚0 proc.cpu.threads‚17( disk.nvme0c0n1.out‚38143.68701171875 disk.nvme1c1n1.in‚0 disk.md0.out‚ 108.25390625! disk.dm-0.out‚42724.072265625( proc.memory.availableMB‚ 492.67578125 cpu‚26.081209145334302) disk./.usagePercent‚76.63284771373695 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme0n1.in‚ 4.3359375 disk.dm-1.in‚0" disk.dm-1.out‚6091.82177734375# proc.memory.rssMB‚ 2844.33984375* proc.memory.percent‚17.360472679138184snäižñ‚(:“ ‰ÕôÒÕÈ´Ì gpu.0.gpu‚66.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚249.871' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚83.29033333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X×AU2ò- _wandbruntime‚4116  _runtime‚4116YG)*Éóà % _timestamp‚1.7844906396983032e+09 _runtime‚4116.273220672 _step‚645 ' lr-AdamW/pg1‚0.00019566892941135652 ' lr-AdamW/pg2‚0.00019566892941135652 trainer/global_step‚7999 ¿áÎôÈ _runtime‚4116.273220672 _step‚645 ' lr-AdamW/pg1‚0.00019566892941135652 ' lr-AdamW/pg2‚0.00019566892941135652 trainer/global_step‚7999 % _timestamp‚1.7844906396983032e+09…£+O2õ- _wandbruntime‚4116  _runtime‚4116Áh|šOöãO % _timestamp‚1.7844906396989746e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80765374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚611077074090496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142955832 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63426053 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚485380539200000 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7665589522432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42327392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80775517 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚611077056571904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80765374 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7651792989828 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 _runtime‚4116.274355864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62190460 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7665248736256 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 _step‚646 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42327390 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚618742305308160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7665589522432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80775517 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21098960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚7999 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚‹™œùš493281677690880 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚611077056571904 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚618742663612928 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚618742305308160 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142965977 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62190458 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7901162915328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚612331973364308 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7651453066506 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚611077074090496 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63426352 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375µÈùî'÷èO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚611077074090496 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63426352 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 % _timestamp‚1.7844906396989746e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80765374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚611077074090496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142955832 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63426053 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚485380539200000 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7665589522432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42327392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80775517 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚611077056571904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80765374 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7651792989828 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 _runtime‚4116.274355864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62190460 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7665248736256 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 _step‚646 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42327390 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚618742305308160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7665589522432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80775517 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21098960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚7999 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚493281677690880 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚611077056571904 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚618742663612928 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚618742305308160 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142965977 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62190458 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7901162915328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚612331973364308 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7651453066506 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260†!âg12ø- _wandbruntime‚4116  _runtime‚4116i,Nó&ùíM FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚618817785300480 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63433973 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚493341508157440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚604753589417600 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80785406 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42332471 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7652381289742 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚611151248340992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42332473 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142983368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62197960 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚612406327670136 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚4116.644155603 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚611151248340992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7902246018048 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 % _timestamp‚1.7844906400689948e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63433676 trainer/global_step‚7999 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7666519440896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚295 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚493341661550592 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142973209 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142983368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚297 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚485439391108096 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0NŽÕ0úòM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7666519440896 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚611151265859584 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142983368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚297 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚485439391108096 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚618817785300480 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63433973 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚493341508157440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚604753589417600 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80785406 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42332471 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7652381289742 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚611151248340992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42332473 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142983368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62197960 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚612406327670136 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚4116.644155603 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚611151248340992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7902246018048 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 % _timestamp‚1.7844906400689948e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63433676 trainer/global_step‚7999 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7666519440896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚295 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚493341661550592 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142973209 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6894211173057556 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20430777966976166 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11562500149011612 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1744610220193863 particle/accuracy_5_of_5‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9983673095703125 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8517094254493713 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1428571492433548 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22099831700325012 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7799999713897705 jet/accuracy_1_of_5‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7059128880500793 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1045110821723938 +particle/specificity‚0.9092379808425903 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_5‚1 epoch‚9 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8421052694320679 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21587224304676056 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9938666224479675 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.49574294686317444 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03493543714284897 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10000000149011612 jet/accuracy_3_of_5‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2777777910232544 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5278645753860474 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.018897512927651405 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18892841041088104 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.21723699569702148 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.19701111316680908 .particle/accuracy_4_of_4‚0.950292706489563 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7030588984489441 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05120718479156494 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5615473389625549 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4658505618572235 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 +jet/accuracy_3_of_3‚0.07522869855165482 *jet/accuracy_2_of_4‚0.6150780320167542 +jet/accuracy_4_of_4‚0.01666666567325592 7validation_average_jet_accuracy‚0.36326974630355835 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11713530868291855 )jet/accuracy_1_of_2‚0.725469708442688 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10000000149011612 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9882606267929077 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18154682219028473 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6633714437484741 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740737318992615 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 particle/accuracy_2_of_4‚1 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 'particle/f_score‚0.6359825730323792 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.28120043873786926 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21705935895442963 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9835749864578247 particle/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9012381434440613 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6706254482269287 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.029911156743764877 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.49744200706481934 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7292296290397644 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.035686563700437546 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988153576850891 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.675601601600647 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16782429814338684 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 jet/accuracy_2_of_5‚1 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3188250958919525 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.15626578032970428 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.036099858582019806 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 particle/accuracy_2_of_2‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1496845781803131 particle/accuracy_3_of_3‚1 /particle/accuracy_3_of_4‚0.9973545074462891 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8574560880661011 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980396032333374 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6966761350631714 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7345117926597595 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9961735010147095 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6530347466468811 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9938672184944153 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8827397227287292 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9453316926956177 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.686495304107666 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.672758936882019 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7077411413192749 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9945982098579407 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8549145460128784 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39838722348213196 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.020299509167671204 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6858464479446411 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8421052694320679 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6786329746246338 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9955058097839355 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8473501801490784 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6810992956161499 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16994278132915497 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.029547246173024178 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5957959294319153 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4469697177410126 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4699999988079071 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7345117926597595 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6332507133483887 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16855232417583466 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1428571492433548 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6718811988830566 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6790270805358887 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6347241401672363 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6120545864105225 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.667910635471344 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +jet/accuracy_2_of_2‚0.15868286788463593 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.949999988079071 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9285714030265808 )loss/val_total_loss‚8.472781069557206 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.985716700553894 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.872823178768158 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4404855966567993 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 trainer/global_step‚8009 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5565363168716431 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3956611752510071 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.020299509167671204 particle/accuracy_1_of_4‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4626867175102234 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2927420139312744 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +particle/sensitivity‚0.6210547685623169 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9899324774742126 _runtime‚4140.913283513 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18311947584152222 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.725986897945404 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_3‚1 _step‚649 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970107078552246 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6354218125343323 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7239317297935486 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.19709289073944092 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6656727194786072 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6759566068649292 % _timestamp‚1.7844906643378356e+09 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6076388955116272 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.640183687210083 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9765211939811707 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9983673095703125 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05‹æ §&„Ì{ 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9953060150146484 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.26964035630226135 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21718499064445496 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03512745723128319 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9925529956817627 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9896722435951233 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7536628842353821 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6711882948875427 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 +jet/accuracy_1_of_1‚0.32814499735832214 +jet/accuracy_3_of_4‚0.09624873101711273 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.771794855594635 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 *jet/accuracy_1_of_4‚0.9505103230476379 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6404943466186523 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.0996013730764389 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.673902153968811 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7049117684364319 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9963349103927612 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1677365005016327 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6946905851364136 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9173877239227295 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6894211173057556 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20430777966976166 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11562500149011612 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1744610220193863 particle/accuracy_5_of_5‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9983673095703125 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8517094254493713 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1428571492433548 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22099831700325012 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7799999713897705 jet/accuracy_1_of_5‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7059128880500793 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1045110821723938 +particle/specificity‚0.9092379808425903 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_5‚1 epoch‚9 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8421052694320679 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21587224304676056 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9938666224479675 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.49574294686317444 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03493543714284897 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10000000149011612 jet/accuracy_3_of_5‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2777777910232544 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5278645753860474 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.018897512927651405 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18892841041088104 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.21723699569702148 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.19701111316680908 .particle/accuracy_4_of_4‚0.950292706489563 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7030588984489441 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05120718479156494 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5615473389625549 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4658505618572235 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 +jet/accuracy_3_of_3‚0.07522869855165482 *jet/accuracy_2_of_4‚0.6150780320167542 +jet/accuracy_4_of_4‚0.01666666567325592 7validation_average_jet_accuracy‚0.36326974630355835 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11713530868291855 )jet/accuracy_1_of_2‚0.725469708442688 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10000000149011612 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9882606267929077 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18154682219028473 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6633714437484741 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740737318992615 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 particle/accuracy_2_of_4‚1 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 'particle/f_score‚0.6359825730323792 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.28120043873786926 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21705935895442963 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9835749864578247 particle/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9012381434440613 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6706254482269287 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.029911156743764877 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.49744200706481934 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7292296290397644 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.035686563700437546 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988153576850891 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.675601601600647 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16782429814338684 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 jet/accuracy_2_of_5‚1 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3188250958919525 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.15626578032970428 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.036099858582019806 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 particle/accuracy_2_of_2‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1496845781803131 particle/accuracy_3_of_3‚1 /particle/accuracy_3_of_4‚0.9973545074462891 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8574560880661011 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980396032333374 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6966761350631714 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7345117926597595 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9961735010147095 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6530347466468811 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9938672184944153 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8827397227287292 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9453316926956177 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.686495304107666 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.672758936882019 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7077411413192749 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9945982098579407 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8549145460128784 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39838722348213196 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.020299509167671204 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6858464479446411 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8421052694320679 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6786329746246338 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9955–±{~+058097839355 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8473501801490784 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6810992956161499 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16994278132915497 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.029547246173024178 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5957959294319153 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4469697177410126 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4699999988079071 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7345117926597595 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6332507133483887 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16855232417583466 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1428571492433548 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6718811988830566 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6790270805358887 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6347241401672363 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6120545864105225 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.667910635471344 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +jet/accuracy_2_of_2‚0.15868286788463593 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.949999988079071 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9285714030265808 )loss/val_total_loss‚8.472781069557206 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.985716700553894 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.872823178768158 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4404855966567993 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 trainer/global_step‚8009 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5565363168716431 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3956611752510071 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.020299509167671204 particle/accuracy_1_of_4‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4626867175102234 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2927420139312744 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +particle/sensitivity‚0.6210547685623169 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9899324774742126 _runtime‚4140.913283513 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18311947584152222 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.725986897945404 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_3‚1 _step‚649 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970107078552246 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6354218125343323 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7239317297935486 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.19709289073944092 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6656727194786072 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6759566068649292 % _timestamp‚1.7844906643378356e+09 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6076388955116272 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.640183687210083 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9765211939811707 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9983673095703125 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.047211937606334686 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.027241062372922897 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7133286595344543 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6714388728141785 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20430777966976166 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16856029629707336 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6711882948875427 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4838328957557678 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08759842813014984 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7799999713897705 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8898454308509827 )jet/accuracy_2_of_3‚0.535253643989563 particle/accuracy_1_of_3‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20215709507465363‰ð^ú¦p…‚(:å ¶ÕôÒ¹ŒáË disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚38336.90576171875 disk.rbd0.out‚ 0.4697265625$ memory_percent‚95.50349712371826# proc.memory.rssMB‚ 2847.69921875 disk.nvme1c1n1.in‚0 disk.md0.out‚ 109.32421875 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚42974.20361328125 disk.dm-1.in‚0" disk.dm-1.out‚6337.62451171875 disk.md0.in‚0 network.recv‚ 1.133548e+06) disk./.usagePercent‚76.63266955787051$ disk./.usageGB‚1176.503059387207( disk.nvme1c1n1.out‚11209.88916015625 disk.nvme1n1.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 736.70703125 network.sent‚ 1.0939648e+07 disk.nvme0c0n1.in‚ 4.3359375* proc.memory.percent‚17.380976676940918 cpu‚26.0894504780875 proc.cpu.threads‚17& disk.nvme1n1.out‚11209.88916015625( disk.nvme0c0n1.out‚38336.90576171875šy¤'ž†‚(:“ ¶ÕôÒ£ºÇÌ gpu.0.gpu‚90.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚252.468' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.15599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_Žú2‡- _wandbruntime‚4162  _runtime‚4162…W³JɈà ' lr-AdamW/pg1‚0.00019543617259033068 ' lr-AdamW/pg2‚0.00019543617259033068 trainer/global_step‚8049 % _timestamp‚1.7844906854347885e+09 _runtime‚4162.009773182 _step‚650éà9Î‰È _step‚650 ' lr-AdamW/pg1‚0.00019543617259033068 ' lr-AdamW/pg2‚0.00019543617259033068 trainer/global_step‚8049 % _timestamp‚1.7844906854347885e+09 _runtime‚4162.009773182ŠÝz¶`2Š- _wandbruntime‚4162  _runtime‚4162À~ðOé'‹ãO WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7736633399296 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42767922 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81434689 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 trainer/global_step‚8049 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62806722 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚624469928292864 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62806722 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64087870 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7722726058306 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚497891163482624 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7973945988096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7736633399296 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚497891321077248 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62806724 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144231268 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7974079158272 _runtime‚4162.014538861 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚489917217494528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81434689 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144241413 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚610268555662320 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚624470286597632 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚616733653198336 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 % _timestamp‚1.7844906854357128e+09 _step‚651 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚617990924757944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81424546 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62806724 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64087586 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7736292613120 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21319948 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚617991281720626 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375ük¢…î'ŒèO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚624470286597632 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚489917241918976 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚610268538622960 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7722386134984 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚624469928292864 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚497891163482624 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7973945988096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7736633399296 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚497891321077248 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62806724 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144231268 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7974079158272 _runtime‚4162.014538861 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚489917217494528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81434689 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144241413 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚610268555662320 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚624470286597632 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚616733653198336 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 % _timestamp‚1.7844906854357128e+09 _step‚651 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚617990924757944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81424546 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62806724 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64087586 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7736292613120 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21319948 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚617991281720626 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7736633399296 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42767922 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81434689 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 trainer/global_step‚8049 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62806722 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚624469928292864 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62806722 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64087870 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7722726058306 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81444578 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62814224 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64095508 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144258804 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚497951153519104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚616807844967424 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7737563317760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7723314358220 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21322215 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7737563317760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚489976093827072 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 _step‚652 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42773001 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7975035267584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚4162.724724335 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144248645 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚616807844967424 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144248645 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚624545408285184 WCDeviceStatsMonitor.on_train_batch_end/=¥M[}active_bytes.large_pool.freed‚616807827448832 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚624545049973248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42773003 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚624545408285184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚616807827448832 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚497950991737344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62814224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62814226 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81444578 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7737222524416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64095216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21322505 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 % _timestamp‚1.7844906861495914e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62814226 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81434421 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7974897910272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62814224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62814226 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81444578 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7737222524416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64095216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21322505 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 % _timestamp‚1.7844906861495914e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62814226 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81434421 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7974897910272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81444578 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62814224 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64095508 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144258804 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚497951153519104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚616807844967424 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7737563317760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7723314358220 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21322215 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7737563317760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚489976093827072 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 _step‚652 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42773001 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7975035267584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚4162.724724335 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144248645 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚616807844967424 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144248645 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚624545408285184 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚616807827448832 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚624545049973248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42773003 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚624545408285184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚616807827448832ŒÙ“/œ2- _wandbruntime‚4162  _runtime‚4162Þì!®‘‚ 0loss/FRt1/assignment_loss‚0.8341420521378046 /loss/FBt1/detection_loss‚0.5051542412695765 %loss/total_loss‚8.923237387646603 2loss/FBt1/assignment_loss‚0.014948408729366387 /loss/FRt1/detection_loss‚1.3563603218019067 /loss/FRt2/detection_loss‚0.5388488782706086 epoch‚10 _runtime‚4162.728417242 2loss/SRqqt1/assignment_loss‚0.5101224106474141 2loss/SRqqt2/assignment_loss‚0.6747406761590499 1loss/SRqqt1/detection_loss‚0.6585667772188751 /loss/FBt2/detection_loss‚0.8339730182455646 % _timestamp‚1.7844906861506772e+09 0loss/FRt2/assignment_loss‚1.8668360576240963 1loss/FBt2/assignment_loss‚0.03361350607483812 0loss/SRqqt2/detection_loss‚1.095931039467503 trainer/global_step‚8049 _step‚653ª°¥j ’‡ 2loss/FBt1/assignment_loss‚0.014948408729366387 /loss/FRt1/detection_loss‚1.3563603218019067 /loss/FRt2/detection_loss‚0.5388488782706086 epoch‚10 _runtime‚4162.728417242 2loss/SRqqt1/assignment_loss‚0.5101224106474141 2loss/SRqqt2/assignment_loss‚0.6747406761590499 1loss/SRqqt1/detection_loss‚0.6585667772188751 /loss/FBt2/detection_loss‚0.8339730182455646 % _timestamp‚1.7844906861506772e+09 0loss/FRt2/assignment_loss‚1.8668360576240963 1loss/FBt2/assignment_loss‚0.03361350607483812 0loss/SRqqt2/detection_loss‚1.095931039467503 trainer/global_step‚8049 _step‚653 0loss/FRt1/assignment_loss‚0.8341420521378046 /loss/FBt1/detection_loss‚0.5051542412695765 %loss/total_loss‚8.923237387646603(­\“‚(:Ñ ÅÕôÒ˜’ÈË disk.nvme1c1n1.in‚0 disk.nvme1c1n1.out‚11251.5& proc.memory.availableMB‚ 736.671875* proc.memory.percent‚17.380976676940918 cpu‚26.087997026389715 disk.md0.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚38352.88232421875& disk.nvme0n1.out‚38352.88232421875$ memory_percent‚95.50371170043945# proc.memory.rssMB‚ 2847.69921875 proc.cpu.threads‚17 network.recv‚ 1.137698e+06) disk./.usagePercent‚76.61719906894999% disk./.usageGB‚1176.2655487060547 disk.nvme1n1.out‚11251.5 disk.dm-0.in‚ 4.3359375" disk.dm-0.out‚42990.9833984375 disk.dm-1.in‚0" disk.dm-1.out‚6377.72607421875 network.sent‚ 1.0997803e+07 disk.nvme1n1.in‚0 disk.md0.out‚ 109.64453125 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375àPßž”‚(:“ ÅÕôÒí©Ì gpu.0.gpu‚90.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚265.295' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.43166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ§ˆŸa•‚(:Ö ÔÕôÒЪÖË network.recv‚ 1.141991e+06% disk.nvme1n1.out‚11457.3330078125 disk.md0.in‚0 disk.md0.out‚ 109.8828125 disk.rbd0.in‚0% disk.nvme0c0n1.out‚38353.69140625$ memory_percent‚95.50371170043945* proc.memory.percent‚17.380976676940918( disk./.usagePercent‚76.6171970811579% disk./.usageGB‚1176.2655181884766 disk.nvme0n1.in‚ 4.3359375# disk.nvme0n1.out‚38353.69140625" disk.dm-0.out‚42991.6162109375& proc.memory.availableMB‚ 736.671875 cpu‚26.086609532031368 network.sent‚ 1.1060983e+07 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚11457.3330078125 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0 disk.dm-1.out‚ 6583.203125 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1n1.in‚0# proc.memory.rssMB‚ 2847.69921875 proc.cpu.threads‚170ž–‚(:“ ÔÕôÒ²ÛÌ gpu.0.gpu‚63.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚255.805' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.26833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒWé2—- _wandbruntime‚4187  _runtime‚4187ŠJo\Ș ' lr-AdamW/pg2‚0.00019520199006877026 trainer/global_step‚8099 $ _timestamp‚1.784490710921039e+09 _runtime‚4187.496047224 _step‚654 ' lr-AdamW/pg1‚0.00019520199006877026HÁ¥Í™Ç trainer/global_step‚8099 $ _timestamp‚1.784490710921039e+09 _runtime‚4187.496047224 _step‚654 ' lr-AdamW/pg1‚0.00019520199006877026 ' lr-AdamW/pg2‚0.00019520199006877026Ž£¥î2š- _wandbruntime‚4187  _runtime‚4187HÛÍ›é'›ãO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8021907280896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚620443224134144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63181824 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7782820434944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 trainer/global_step‚8099 % _timestamp‚1.7844907109220884e+09 _runtime‚4187.497461226 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚620443241652736 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7783161221120 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7768829199992 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚492861034121728 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8022036256768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚613939842746960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚628226044569088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚500882941402624 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63181822 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7769169123314 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43021970 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 _step‚655 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81930291 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚628226044569088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚500882941402624 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63181822 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7769169123314 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43021970 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 _step‚655õbÏô2- _wandbruntime‚4187  _runtime‚4187Ö¤µ¬ó&žíM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8099 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7784091139584 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 145119347 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚620517433421824 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7783750346240 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63189324 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7783750346240 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚621783026252668 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 _step‚656 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 % _timestamp‚1.7844907112975688e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚628301524561408 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚620517415903232 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43027051 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81930023 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63189324 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63189326 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81940180 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64476040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21449278 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8022996326912 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8022863163904 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚614013268829440 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚628301524561408 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚500942931205632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7770097346662 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚492919910454272 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145129506 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚621783383215462 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145129506 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 145119347 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚500942773618176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63189326 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚620517433421824 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81930023 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚492919934878720 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7769757423228 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7784091139584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21448989 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚620517415903232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375§Ejø&ŸòM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133163008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 % _timestamp‚1.7844907112975688e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚628301524561408 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚620517415903232 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43027051 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81930023 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63189324 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63189326 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81940180 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64476040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21449278 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8022996326912 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8022863163904 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚614013268829440 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚628301524561408 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚500942931205632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7770097346662 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚492919910454272 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145129506 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚621783383215462 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145129506 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 145119347 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚500942773618176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63189326 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚620517433421824 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81930023 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚492919934878720 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7769757423228 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7784091139584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21448989 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚620517415903232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8099 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7784091139584 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 145119347 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚620517433421824 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7783750346240 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63189324 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7783750346240 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚621783026252668 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 _step‚656 Eõ¿ã ¡„ 1loss/SRqqt1/detection_loss‚0.6510447830767996 /loss/FBt1/detection_loss‚0.4349838913548127 2loss/SRqqt1/assignment_loss‚0.6025091636398158 2loss/SRqqt2/assignment_loss‚0.6776529984250802 1loss/SRqqt2/detection_loss‚1.0078166892943794 trainer/global_step‚8099 _step‚657 3loss/FBt2/assignment_loss‚0.0002948156332232246 /loss/FRt1/detection_loss‚1.4774444178094142 /loss/FRt2/detection_loss‚0.5271579986630787 % _timestamp‚1.7844907112986622e+09 _runtime‚4187.873369308 /loss/FBt2/detection_loss‚0.4953545153968864 %loss/total_loss‚8.676103491693413 epoch‚10 /loss/FRt1/assignment_loss‚0.896517300077023 0loss/FRt2/assignment_loss‚1.8473434158211923 2loss/FBt1/assignment_loss‚0.057983502501705676iêá’¢‰ /loss/FBt2/detection_loss‚0.4953545153968864 %loss/total_loss‚8.676103491693413 epoch‚10 /loss/FRt1/assignment_loss‚0.896517300077023 0loss/FRt2/assignment_loss‚1.8473434158211923 2loss/FBt1/assignment_loss‚0.057983502501705676 1loss/SRqqt1/detection_loss‚0.6510447830767996 /loss/FBt1/detection_loss‚0.4349838913548127 2loss/SRqqt1/assignment_loss‚0.6025091636398158 2loss/SRqqt2/assignment_loss‚0.6776529984250802 1loss/SRqqt2/detection_loss‚1.0078166892943794 trainer/global_step‚8099 _step‚657 3loss/FBt2/assignment_loss‚0.0002948156332232246 /loss/FRt1/detection_loss‚1.4774444178094142 /loss/FRt2/detection_loss‚0.5271579986630787 % _timestamp‚1.7844907112986622e+09 _runtime‚4187.873369308‘ð01Hl£‚(:á ãÕôÒª‘ÌË disk.md0.in‚0 disk.dm-1.out‚ 6583.31640625# proc.memory.rssMB‚ 2847.69921875 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚38383.95751953125( disk.nvme1c1n1.out‚11458.25537109375 disk.dm-1.in‚0 disk.rbd0.in‚0 cpu‚26.086539583383782 network.recv‚ 1.14515e+06 disk.md0.out‚ 110.21484375 disk.dm-0.in‚ 4.3359375" disk.dm-0.out‚43021.9716796875 disk.rbd0.out‚ 0.4697265625' proc.memory.availableMB‚ 737.8984375* proc.memory.percent‚17.380976676940918 disk.nvme1n1.in‚0 network.sent‚ 1.1063905e+07) disk./.usagePercent‚76.62075970153859( disk.nvme0c0n1.out‚38383.95751953125 disk.nvme1c1n1.in‚0$ memory_percent‚95.49622535705566$ disk./.usageGB‚1176.320213317871& disk.nvme1n1.out‚11458.25537109375òìØž¤‚(:“ ãÕôÒýìºË gpu.0.gpu‚61.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚264.686' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.22866666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k­,2¥- _wandbruntime‚4212  _runtime‚4212ÍYÃ%ǦÁ & lr-AdamW/pg1‚0.0001949663855975836 & lr-AdamW/pg2‚0.0001949663855975836 trainer/global_step‚8149 % _timestamp‚1.7844907361899495e+09 _runtime‚4212.765082388 _step‚658?s9Ì§Æ & lr-AdamW/pg1‚0.0001949663855975836 & lr-AdamW/pg2‚0.0001949663855975836 trainer/global_step‚8149 % _timestamp‚1.7844907361899495e+09 _runtime‚4212.765082388 _step‚658’‡ŸáR2¨- _wandbruntime‚4212  _runtime‚4212=¦xG©ãO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚631982160845312 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63556922 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64849046 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7829689042944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8069986594304 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174´=Ž•¢"720 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82415750 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7829348256768 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚631982519150080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚503874714658816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚625426776098642 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 _step‚659 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚624152830107136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43276020 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚495804850748928 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 145972672 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63556922 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚625426419135960 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚8149 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚624152830107136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚495804875173376 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚631982160845312 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚624152812588544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968‘r¡ùî'ªèO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8069986594304 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82415750 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7829348256768 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚631982519150080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚503874714658816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚625426776098642 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 _step‚659 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚624152830107136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43276020 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚495804850748928 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 145972672 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63556922 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚625426419135960 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚8149 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚624152830107136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚495804875173376 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚631982160845312 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚624152812588544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚631982160845312 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63556922 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64849046 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7829689042944“#ã 2«- _wandbruntime‚4213  _runtime‚421356ÄAó&¬íM ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚617684572953440 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21575839 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚624227004357632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8070948605952 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4213.132151528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚624227021876224 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146000208 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚632057640837632 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7830278168064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43281101 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8070828025856 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 146000208 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63564424 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚632057282525696 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚503934700111872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚617684589992800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7830618961408 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64856663 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7830278168064 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7816200488236 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7816540411670 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63564424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 120580096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚625500773441676 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 145990049 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82435782 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82425625 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82425625 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 145004544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚8149 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82425625 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 145004544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚8149 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4213.132151528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚624227021876224 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146000208 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚632057640837632 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7830278168064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43281101 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8070828025856 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 146000208 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63564424 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚632057282525696 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚503934700111872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚617684589992800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7830618961408 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64856663 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7830278168064 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7816200488236 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7816540411670 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63564424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 120580096”¿z³Õ2®- _wandbruntime‚4213  _runtime‚4213g;ÓÞ¯ trainer/global_step‚8149 _runtime‚ 4213.13321295 /loss/FRt1/detection_loss‚1.5114607473997654 1loss/SRqqt2/detection_loss‚0.8824537343959526 /loss/FBt1/detection_loss‚0.5161789108640873 .loss/FBt2/detection_loss‚0.788648114229242 % _timestamp‚1.7844907365585234e+09 0loss/FRt1/assignment_loss‚0.7547509892847528 2loss/SRqqt1/assignment_loss‚0.4942919662509245 2loss/FBt2/assignment_loss‚0.017424384562661526 /loss/FRt2/detection_loss‚0.5303019004677403 %loss/total_loss‚8.638399968928717 0loss/FRt2/assignment_loss‚1.8850464071655681 1loss/SRqqt2/assignment_loss‚0.622232273879043 _step‚661 2loss/FBt1/assignment_loss‚0.036193783501816006 1loss/SRqqt1/detection_loss‚0.5994167569271638 epoch‚10sòò °† 0loss/FRt2/assignment_loss‚1.8850464071655681 1loss/SRqqt2/assignment_loss‚0.622232273879043 _step‚661 2loss/FBt1/assignment_loss‚0.036193783501816006 1loss/SRqqt1/detection_loss‚0.5994167569271638 epoch‚10 trainer/global_step‚8149 _runtime‚ 4213.13321295 /loss/FRt1/detection_loss‚1.5114607473997654 1loss/SRqqt2/detection_loss‚0.8824537343959526 /loss/FBt1/detection_loss‚0.5161789108640873 .loss/FBt2/detection_loss‚0.788648114229242 % _timestamp‚1.7844907365585234e+09 0loss/FRt1/assignment_loss‚0.7547509892847528 2loss/SRqqt1/assignment_loss‚0.4942919662509245 2loss/FBt2/assignment_loss‚0.017424384562661526 /loss/FRt2/detection_loss‚0.5303019004677403 %loss/total_loss‚8.638399968928717•Ó^7Xn±‚(:ã òÕôÒíà—Ì* proc.memory.percent‚17.380976676940918 cpu‚26.087643776998238& disk.nvme0n1.out‚38420.36572265625" disk.dm-0.out‚43058.1787109375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚11459.068359375$ disk.nvme1n1.out‚11459.068359375( proc.memory.availableMB‚ 738.06640625 network.sent‚ 1.1139171e+07% disk./.usageGB‚1176.5689811706543" disk.dm-1.out‚6583.46728515625 disk.nvme1n1.in‚0 disk.md0.out‚ 110.61328125# proc.memory.rssMB‚ 2847.69921875 proc.cpu.threads‚17 network.recv‚ 1.149538e+06 disk.dm-0.in‚ 4.3359375 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375$ memory_percent‚95.49520015716553) disk./.usagePercent‚76.63696343726782( disk.nvme0c0n1.out‚38420.36572265625 disk.md0.in‚0v9[Æž²‚(:“ òÕôÒáÎÌ gpu.0.gpu‚91.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚215.811' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚71.93700000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Òl{d³‚(:Ù ÖôÒè—êË disk.nvme1n1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 737.81640625* proc.memory.percent‚17.380976676940918) disk./.usagePercent‚76.63279280098034% disk./.usageGB‚1176.5049514770508 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 11516.8671875$ memory_percent‚95.49672603607178# proc.memory.rssMB‚ 2847.69921875 cpu‚26.088693483665512 proc.cpu.threads‚17 network.recv‚ 1.153981e+06% disk.nvme0c0n1.out‚38563.89453125# disk.nvme0n1.out‚38563.89453125" disk.nvme1n1.out‚ 11516.8671875 disk.md0.out‚ 110.9453125 disk.dm-0.in‚ 4.3359375" disk.dm-1.out‚6583.56103515625 disk.rbd0.out‚ 0.4697265625 network.sent‚ 1.1189009e+07 disk.md0.in‚0# disk.dm-0.out‚43258.68212890625 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375¿f#X“´‚(:ˆ ÖôÒšÌÜË gpu.0.gpu‚91.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚257.313' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚85.771 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð92µ- _wandbruntime‚4237  _runtime‚4237ï¾Êɶà ' lr-AdamW/pg2‚0.00019472936295045456 trainer/global_step‚8199 % _timestamp‚1.7844907613629432e+09 _runtime‚4237.938017643 _step‚662 ' lr-AdamW/pg1‚0.00019472936295045456(ö Î·È _step‚662 ' lr-AdamW/pg1‚0.00019472936295045456 ' lr-AdamW/pg2‚0.00019472936295045456 trainer/global_step‚8199 % _timestamp‚1.7844907613629432e+09 _runtime‚4237.938017643–•LG2¸- _wandbruntime‚4237  _runtime‚4237ƒœ«Œé'¹ãO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43530070 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82911352 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _runtime‚4237.939969677 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82921495 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63932022 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚627862401042944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63932022 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚635738277121536 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚627862418561536 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43530072 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚305 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚506866548212224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82911352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65231330 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21700953 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63932024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚498748667376128 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82921495 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8118011909120 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚621282468034320 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7861715330008 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63932024 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7875876078592 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚307 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚629144166324968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚627862418561536 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7876216864768 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚621282450994960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 trainer/global_step‚8199 _step‚663 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146853519 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8117880836096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚635738635426304 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚627862401042944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65231023 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21701258 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚635738635426304 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚506866703709696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚635738277121536 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7876216864768 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚498748691800576î´MPî'ºèO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚621282468034320 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7861715330008 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63932024 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7875876078592 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚307 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚629144166324968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚627862418561536 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7876216864768 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚621282450994960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 trainer/global_step‚8199 _step‚663 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146853519 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8117880836096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚635738635426304 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚627862401042944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65231023 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21701258 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚635738635426304 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚506866703709696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚635738277121536 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7876216864768 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚498748691800576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43530070 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82911352 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _runtime‚4237.939969677 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82921495 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63932022 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚627862401042944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63932022 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚635738277121536 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚627862418561536 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43530072 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚305 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚506866548212224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82911352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65231330 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21700953 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63932024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚498748667376128 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82921495 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8118011909120 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375—Qõá2»- _wandbruntime‚4238  _runtime‚4238ô'ft¼ìM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8118836538880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7862983476678 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82931384 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8118967604736 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146860751 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 _h_“~"KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚498807543708672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146870910 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7877146783232 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21703515 _runtime‚4238.300118193 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 _step‚664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7877146783232 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚302 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63939526 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚635813757113856 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82931384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65238968 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65238666 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21703815 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82921227 trainer/global_step‚8199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43535151 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7876805989888 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚621355877077440 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚627936610330624 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚506926535737856 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚627936610330624 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63939524 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43535153 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82921227 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚627936592812032 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚627936592812032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7862643553244 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 $ _timestamp‚1.784490761724856e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63939526 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚635813757113856 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146860751 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚498807568133120 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚621355894116800 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63939524 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚300 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304³oËÁ÷&½ñM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43535153 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82921227 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚627936592812032 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚627936592812032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7862643553244 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 $ _timestamp‚1.784490761724856e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63939526 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚635813757113856 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146860751 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚498807568133120 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚621355894116800 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63939524 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚300 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8118836538880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7862983476678 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82931384 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8118967604736 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146860751 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚498807543708672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146870910 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7877146783232 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21703515 _runtime‚4238.300118193 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 _step‚664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7877146783232 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚302 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63939526 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚635813757113856 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82931384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65238968 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65238666 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21703815 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82921227 trainer/global_step‚8199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43535151 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7876805989888 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚621355877077440 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚627936610330624 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚506926535737856 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚627936610330624 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63939524 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240˜͇^>2¾- _wandbruntime‚4238  _runtime‚4238‚puR ¿… 0loss/FRt1/assignment_loss‚0.8563348075005576 2loss/SRqqt2/assignment_loss‚0.4926740369786525 trainer/global_step‚8199 % _timestamp‚1.7844907617259603e+09 _step‚665 /loss/FRt2/detection_loss‚0.6036026436133819 /loss/FBt1/detection_loss‚0.5868688191230033 epoch‚10 /loss/FBt2/detection_loss‚0.8482049067815145 0loss/FRt2/assignment_loss‚1.8719976771505256 2loss/SRqqt1/assignment_loss‚0.5591918386997883 2loss/FBt2/assignment_loss‚5.28996597031058e-05 /loss/FRt1/detection_loss‚1.4109826841537019 1loss/SRqqt1/detection_loss‚0.6331764774045239 1loss/SRqqt2/detection_loss‚0.9608556193665468 3loss/FBt1/assignment_loss‚0.0022265951168144353 %loss/total_loss‚8.826169005548714 _runtime‚4238.300724914Aé'ÀŠ 2loss/SRqqt2/assignment_loss‚0.4926740369786525 trainer/global_step‚8199 % _timestamp‚1.7844907617259603e+09 _step‚665 /loss/FRt2/detection_loss‚0.6036026436133819 /loss/FBt1/detection_loss‚0.5868688191230033 epoch‚10 /loss/FBt2/detection_loss‚0.8482049067815145 0loss/FRt2/assignment_loss‚1.8719976771505256 2loss/SRqqt1/assignment_loss‚0.5591918386997883 2loss/FBt2/assignment_loss‚5.28996597031058e-05 /loss/FRt1/detection_loss‚1.4109826841537019 1loss/SRqqt1/detection_loss‚0.6331764774045239 1loss/SRqqt2/detection_loss‚0.9608556193665468 3loss/FBt1/assignment_loss‚0.0022265951168144353 %loss/total_loss‚8.826169005548714 _runtime‚4238.300724914 0loss/FRt1/assignment_loss‚0.8563348075005576™ý‡MiÁ‚(:Þ ÖôÒÁŒÆË disk.rbd0.out‚ 0.4697265625$ disk.nvme0n1.out‚38564.849609375* proc.memory.percent‚17.380976676940918 cpu‚26.089382628933603 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0 proc.cpu.threads‚17 network.sent‚ 1.1193875e+07% disk./.usageGB‚1176.5206604003906& disk.nvme0c0n1.out‚38564.849609375( disk.nvme1c1n1.out‚11517.50634765625# proc.memory.rssMB‚ 2847.69921875 network.recv‚ 1.157295e+06 disk.nvme1n1.in‚0& disk.nvme1n1.out‚11517.50634765625! disk.dm-1.out‚6583.7255859375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375$ memory_percent‚95.49541473388672% proc.memory.availableMB‚ 738.03125( disk./.usagePercent‚76.6338160169608 disk.md0.out‚ 111.2734375# disk.dm-0.out‚43259.40478515625 disk.dm-1.in‚0®þd‚(:’ ÖôÒ²ç¥Ì gpu.0.gpu‚57.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚240.59' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚80.19666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®zÎdkÂ(:à ŸÖôÒ‚½ÒË network.recv‚ 1.161288e+06' disk.nvme0c0n1.out‚38634.9599609375% disk.nvme0n1.out‚38634.9599609375$ memory_percent‚95.49541473388672% proc.memory.availableMB‚ 738.03125* proc.memory.percent‚17.380976676940918# disk.dm-0.out‚43329.90966796875 disk.dm-1.in‚0! disk.dm-1.out‚6583.8154296875 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚11518.60009765625 disk.md0.in‚0 disk.md0.out‚ 111.55078125 cpu‚26.0893601482448 network.sent‚ 1.1256249e+07 disk.nvme0n1.in‚ 4.3359375( disk.nvme1c1n1.out‚11518.60009765625 disk.nvme1n1.in‚0# proc.memory.rssMB‚ 2847.69921875 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 4.3359375) disk./.usagePercent‚76.65245877358953% disk./.usageGB‚1176.8068733215332 disk.dm-0.in‚ 4.3359375 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625ëŒ1“Ä‚(:ˆ ŸÖôÒàö Ì gpu.0.gpu‚94.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚268.335' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚89.445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ƶ ý2Å- _wandbruntime‚4263  _runtime‚4263‰•`ÇÆÁ _step‚666 ' lr-AdamW/pg1‚0.00019449092592378197 ' lr-AdamW/pg2‚0.00019449092592378197 trainer/global_step‚8249 $ _timestamp‚1.784490786737005e+09 _runtime‚ 4263.31201333•½ÌÇÆ _runtime‚ 4263.31201333 _step‚666 ' lr-AdamW/pg1‚0.00019449092592378197 ' lr-AdamW/pg2‚0.00019449092592378197 trainer/global_step‚8249 $ _timestamp‚1.784490786737005e+09šDAEƒ2È- _wandbruntime‚4263  _runtime‚4263^÷Ðá#ÉãO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚639494393397760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚639494393397760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43784120 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚509858505217536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7922403900416 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83417097 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚631572007015936 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65612338 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21828218 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7908498318338 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64307124 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43784122 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚624953772158320 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 _step‚667 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚631571989497344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64307124 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚632861913513976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 OEDeviceStatsMonitor.on_train_batch_start/allocation.ƒ5Ë·small_pool.current‚10143 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 147714076 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64307122 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚509858353914368 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43784122 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚624953772158320 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 _step‚667 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚631571989497344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64307124 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚632861913513976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 147714076 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64307122 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚509858353914368 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚639494393397760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚639494393397760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43784120 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚509858505217536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7922403900416 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83417097 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚631572007015936 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375›06ÜÉ2Ë- _wandbruntime‚4263  _runtime‚4263N‘àÏó&ÌíM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43789203 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65619917 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7923333811712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64314626 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7909426541686 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7923333811712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚631646198785024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚509918339998208 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7923674605056 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚632936624782486 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚631646181266432 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 147731453 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚631646198785024 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚509918188702208 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147741612 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8166828366336 trainer/global_step‚8249 % _timestamp‚1.7844907870992968e+09 _runtime‚4263.674526373 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64314624 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83426986 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43789201 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚625027198240800 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚631646181266432 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7923674605056 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65620186 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚639569873390080 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚632936267819692 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚639569515078144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83426986 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚501751360335872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 _step‚668 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64314624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147741612 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21830716 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8166955237888 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83416829 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚501751384760320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147731453 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83416829 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21830983 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚639569873390080?wkø&ÍòM YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65620186 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚639569873390080 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚632936267819692 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚639569515078144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83426986 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚501751360335872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 _step‚668 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64314624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147741612 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21830716 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8166955237888 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83416829 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚501751384760320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147731453 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83416829 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21830983 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚639569873390080 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43789203 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65619917 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7923333811712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64314626 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7909426541686 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7923333811712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚631646198785024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚509918339998208 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7923674605056 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚632936624782486 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚631646181266432 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 147731453 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚631646198785024 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚509918188702208 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147741612 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8166828366336 trainer/global_step‚8249 % _timestamp‚1.7844907870992968e+09 _runtime‚4263.674526373 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64314624 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83426986 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43789201 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚625027198240800 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚631646181266432 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7923674605056 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935œ¬¯w2Î- _wandbruntime‚4263  _runtime‚4263<‘TD χ 0loss/FRt1/assignment_loss‚0.8150894075023885 0loss/FRt2/assignment_loss‚1.8886656297166402 /loss/FBt2/detection_loss‚1.3467469955721754 trainer/global_step‚8249 _step‚669 2loss/SRqqt2/assignment_loss‚0.6182047870060835 4loss/FBt2/assignment_loss‚0.00023800893546122026 epoch‚10 _runtime‚4263.674990364 2loss/SRqqt1/assignment_loss‚0.5265748425722898 4loss/FBt1/assignment_loss‚0.00012614188952132725 1loss/SRqqt1/detection_loss‚0.5729116068805258 /loss/FBt1/detection_loss‚0.5675450353014436 % _timestamp‚1.7844907871002889e+09 .loss/FRt1/detection_loss‚1.384201018350693 /loss/FRt2/detection_loss‚0.5403716587108127 1loss/SRqqt2/detection_loss‚0.9231352461081359 %loss/total_loss‚9.183810378546172ħéÄvÐŒ 2loss/SRqqt1/assignment_loss‚0.5265748425722898 4loss/FBt1/assignment_loss‚0.00012614188952132725 1loss/SRqqt1/detection_loss‚0.5729116068805258 /loss/FBt1/detection_loss‚0.5675450353014436 % _timestamp‚1.7844907871002889e+09 .loss/FRt1/detection_loss‚1.384201018350693 /loss/FRt2/detection_loss‚0.5403716587108127 1loss/SRqqt2/detection_loss‚0.9231352461081359 %loss/total_loss‚9.183810378546172 0loss/FRt1/assignment_loss‚0.8150894075023885 0loss/FRt2/assignment_loss‚1.8886656297166402 /loss/FBt2/detection_loss‚1.3467469955721754 trainer/global_step‚8249 _step‚69aÔœ69 2loss/SRqqt2/assignment_loss‚0.6182047870060835 4loss/FBt2/assignment_loss‚0.00023800893546122026 epoch‚10 _runtime‚4263.674990364 (lWÑ‚(:Ì ®ÖôÒ’îÓË% proc.memory.availableMB‚ 738.65625& disk.nvme1n1.out‚11546.83740234375 disk.nvme0c0n1.in‚ 4.3359375! disk.nvme0c0n1.out‚ 38798.6875( disk.nvme1c1n1.out‚11546.83740234375* proc.memory.percent‚17.380976676940918 network.recv‚ 1.164625e+06 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0 disk.dm-1.out‚ 6584.0625 disk.nvme0n1.out‚ 38798.6875# memory_percent‚95.4916000366211# proc.memory.rssMB‚ 2847.69921875 cpu‚26.088289741628063 network.sent‚ 1.1262158e+07# disk./.usageGB‚1176.74267578125# disk.dm-0.out‚43520.83447265625 disk.nvme1c1n1.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625) disk./.usagePercent‚76.64827720444552 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 111.9140625 disk.nvme0n1.in‚ 4.3359375lWÒ‚(:’ ®ÖôÒÔéÅË gpu.0.gpu‚91.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚155.67' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚51.88999999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-‰Hô2Ó- _wandbruntime‚4287  _runtime‚4287Dâ4ÈÔ ' lr-AdamW/pg1‚0.00019425107833661873 ' lr-AdamW/pg2‚0.00019425107833661873 trainer/global_step‚8299 % _timestamp‚1.7844908111706543e+09 _runtime‚ 4287.74556915 _step‚670Ù PÍÕÇ ' lr-AdamW/pg2‚0.00019425107833661873 trainer/global_step‚8299 % _timestamp‚1.7844908111706543e+09 _runtime‚ 4287.74556915 _step‚670 ' lr-AdamW/pg1‚0.00019425107833661873ž±ã+2Ö- _wandbruntime‚4287  _runtime‚4287 ^~€è'×âO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 142914560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44038172 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21955308 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 118490112 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7954601460024 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚643250867978752 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚628625076282320 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64682224 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚643250509673984 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8213890387456 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 $ _timestamp‚1.784490811171327e+09 _runtime‚4287.746129711 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚643250509673984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚635281595470336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83902556 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83902556 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64682224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7969272508416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2|½í'ØçO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚643250867978752 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65993478 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚628625059242960 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148584778 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7968931722240 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚635281577951744 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8214008877568 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65993748 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚635281595470336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚643250509673984 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8213890387456 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 $ _timestamp‚1.784490811171327e+09 _runtime‚4287.746129711 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚643250509673984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚635281595470336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83902556 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83902556 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64682224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7969272508416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 142914560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44038172 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21955308 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 118490112 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7954601460024 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚643250867978752 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚628625076282320 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64682224 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44043253 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚635355769720832 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8214845578240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7969861633536 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148612314 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚643325631354368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚636654015008700 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚504695201387520 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8214974546944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚628698502364800 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚643325989666304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148612314 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66001016 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21958043 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64689724 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚635355787239424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66001296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚6366543719714947Ê)ÛòM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚643325989666304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.pŸýú7Ï%eak‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148612314 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66001016 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21958043 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64689724 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚635355787239424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66001296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚636654371971494 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83912431 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44043253 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚635355769720832 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8214845578240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7969861633536 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148612314 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚643325631354368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚636654015008700 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚504695201387520 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8214974546944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚628698502364800 xHå£2Ü- _wandbruntime‚4288  _runtime‚4288 ”†Ý 2loss/SRqqt1/assignment_loss‚0.5239103492539422 3loss/FBt1/assignment_loss‚0.0005278935526550502 3loss/FBt2/assignment_loss‚0.0012248155082540773 epoch‚10 _step‚673 0loss/FRt1/assignment_loss‚0.8488962075731776 /loss/FRt2/detection_loss‚0.6073710495179471 0loss/SRqqt1/detection_loss‚0.580305741298441 trainer/global_step‚8299 0loss/FRt2/assignment_loss‚1.8949803348865595 /loss/FRt1/detection_loss‚1.5088875095521495 1loss/SRqqt2/detection_loss‚0.8691746168096975 .loss/FBt1/detection_loss‚0.531593153718859 _runtime‚4288.107712193 2loss/SRqqt2/assignment_loss‚0.8018440751976608 .loss/FBt2/detection_loss‚1.154038421312968 $loss/total_loss‚9.32275416818231 $ _timestamp‚1.784490811530925e+09Å*¢5 Þ† 0loss/FRt2/assignment_loss‚1.8949803348865595 /loss/FRt1/detection_loss‚1.5088875095521495 1loss/SRqqt2/detection_loss‚0.8691746168096975 .loss/FBt1/detection_loss‚0.531593153718859 _runtime‚4288.107712193 2loss/SRqqt2/assignment_loss‚0.8018440751976608 .loss/FBt2/detection_loss‚1.154038421312968 $loss/total_loss‚9.32275416818231 $ _timestamp‚1.784490811530925e+09 2loss/SRqqt1/assignment_loss‚0.5239103492539422 3loss/FBt1/assignment_loss‚0.0005278935526550502 3loss/FBt2/assignment_loss‚0.0012248155082540773 epoch‚10 _step‚673 0loss/FRt1/assignment_loss‚0.8488962075731776 /loss/FRt2/detection_loss‚0.6073710495179471 0loss/SRqqt1/detection_loss‚0.580305741298441 trainer/global_step‚8299¡Æw{nß‚(:ã ½ÖôÒÙõÆË disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚11550.14892578125 disk.md0.in‚0 disk.md0.out‚ 112.37890625 disk.dm-1.out‚ 6584.14453125 disk.rbd0.in‚0# proc.memory.rssMB‚ 2847.69921875 network.sent‚ 1.1323493e+07 disk.nvme0n1.in‚ 4.3359375( disk.nvme1c1n1.out‚11550.14892578125 disk.rbd0.out‚ 0.4697265625* proc.memory.percent‚17.380976676940918 cpu‚26.088099473413454% disk.nvme0n1.out‚38808.0205078125 disk.nvme1n1.in‚0 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚43532.38330078125 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 4.3359375( proc.memory.availableMB‚ 492.40234375 proc.cpu.threads‚17 network.recv‚ 1.168746e+06) disk./.usagePercent‚76.63279553419447% disk./.usageGB‚1176.5049934387207' disk.nvme0c0n1.out‚38808.0205078125$ memory_percent‚96.99461460113525ÐÈœ÷žà‚(:“ ½ÖôÒø‘¨Ì gpu.0.gpu‚90.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚263.539' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.84633333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}’Sgá‚(:Ü ÌÖôÒ–žõË cpu‚26.088309854053815( disk./.usagePercent‚76.6327980189346 disk.md0.out‚ 112.91796875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375$ disk.nvme0n1.out‚38811.904296875 disk.nvme1c1n1.in‚0* proc.memory.percent‚17.380976676940918 disk.md0.in‚0 disk.rbd0.out‚ 0.4697265625& disk.nvme0c0n1.out‚38811.904296875$ memory_percent‚96.99616432189941# proc.memory.rssMB‚ 2847.69921875 proc.cpu.threads‚17 network.sent‚ 1.1387788e+07% disk./.usageGB‚1176.5050315856934 disk.dm-0.in‚ 4.3359375" disk.dm-0.out‚43535.7900390625! disk.dm-1.out‚6584.3291015625 disk.nvme0n1.in‚ 4.3359375 disk.nvme1n1.in‚0 network.recv‚ 1.173219e+06 disk.dm-1.in‚0& disk.nvme1c1n1.out‚11550.990234375$ disk.nvme1n1.out‚11550.990234375' proc.memory.availableMB‚ 492.1484375 ˆ3“â‚(:ˆ ÌÖôÒÏÊæË gpu.0.gpu‚70.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚251.769' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚83.923 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-ð 2ã- _wandbruntime‚4312  _runtime‚4312O¯9¦ÇäÁ _runtime‚4312.452707161 _step‚674 & lr-AdamW/pg1‚0.0001940098240306108 & lr-AdamW/pg2‚0.0001940098240306108 trainer/global_step‚8349 % _timestamp‚1.7844908358777118e+09nKbÌåÆ _step‚674 & lr-AdamW/pg1‚0.0001940098240306108 & lr-AdamW/pg2‚0.0001940098240306108 trainer/global_step‚8349 % _timestamp‚1.7844908358777118e+09 _runtime‚4312.452707161¢±i²2æ- _wandbruntime‚4312  _runtime‚4312¸P$Œè'çâO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 116392960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚507580117257728 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8001384448354 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚632296363366960 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84398158 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚515842024287744 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8001044525032 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8015800330240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚507580141682176 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65057324 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65057324 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8015800330240 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44292222 trainer/global_step‚8349 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65057322 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84408301 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚675 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8015459544064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚647006625950208 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65057322 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚632296380406320 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66374782 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 149455480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22082560 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66374500 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚640297407891992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚638991183924736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚647006625950208 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84408301 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚638991183924736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚515842165105152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 140817408 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8262023422976 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8015459544064 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44292220 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚640297764854674 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149465625 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚647006984254976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84398158 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8261907030016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.784490835878588e+09 _runtime‚4312.456369667 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚647006984254976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚638991166406144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚647006984254976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84398158 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8261907030016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.784490835878588e+09 _runtime‚4312.456369667 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚647006984254976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚638991166406144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44292222 trainer/global_step‚8349 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65057322 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84408301 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚675 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8015459544064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚647006625950208 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65057322 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚632296380406320 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66374782 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 149455480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22082560 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66374500 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚640297407891992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚638991183924736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 UGDeviceStatsMonitor.on_train_batch_start/requ^õí¿Hested_bytes.small_pool.peak‚ 538001087 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚647006625950208 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84408301 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚638991183924736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚515842165105152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 140817408 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8262023422976 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8015459544064 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44292220 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚640297764854674 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260£TÅ?Ä2é- _wandbruntime‚4312  _runtime‚4312%Ühuó&êíM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84408033 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚639065375693824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22085089 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚515902003244544 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84418190 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149472857 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 _runtime‚4312.814922469 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65064824 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8262985229824 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚632369789449440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚639065358175232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚639065358175232 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚640372119160502 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚632369806488800 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8016730248704 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65064824 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149483016 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84418190 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚647081747630592 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149472857 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚639065375693824 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚647082105942528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22084813 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8262866746880 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65064826 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚640371762197708 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8016389455360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 118482944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66382114 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 trainer/global_step‚8349 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚647082105942528 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8016389455360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8002312671702 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675=„õø&ëòM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚647081747630592 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149472857 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚639065375693824 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚647082105942528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22084813 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8262866746880 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65064826 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚640371762197708 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8016389455360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 118482944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66382114 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 trainer/global_step‚8349 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚647082105942528 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8016389455360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8002312671702 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84408033 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚639065375693824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22085089 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚515902003244544 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84418190 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149472857 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 _runtime‚4312.814922469 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65064824 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8262985229824 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚632369789449440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚639065358175232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚639065358175232 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚640372119160502 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚632369806488800 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8016730248704 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65064824 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149483016 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84418190¤È\”2ì- _wandbruntime‚4312  _runtime‚4312 wD4í€ 0loss/FRt2/assignment_loss‚1.7180384598228722 1loss/SRqqt1/detection_loss‚0.5368471994164533 1loss/SRqqt2/detection_loss‚0.8961128189459743 # _timestamp‚1.78449083624074e+09 /loss/FBt2/detection_loss‚1.0578827445528336 epoch‚10 /loss/FRt1/assignment_loss‚0.841700055545434 1loss/FBt1/assignment_loss‚0.04430112835708542 /loss/FRt1/detection_loss‚1.3490053003608369 /loss/FBt1/detection_loss‚0.5688899185430879 _step‚677 %loss/total_loss‚8.731205551329912 2loss/SRqqt1/assignment_loss‚0.5175692352029676 2loss/SRqqt2/assignment_loss‚0.6535581372373178 3loss/FBt2/assignment_loss‚0.0011665353740676818 /loss/FRt2/detection_loss‚0.5461340179709815 trainer/global_step‚8349 _runtime‚ 4312.81538095™8ï= î… epoch‚10 /loss/FRt1/assignment_loss‚0.841700055545434 1loss/FBt1/assignment_loss‚0.04430112835708542 /loss/FRt1/detection_loss‚1.3490053003608369 /loss/FBt1/detection_loss‚0.5688899185430879 _step‚677 %loss/total_loss‚8.731205551329912 2loss/SRqqt1/assignment_loss‚0.5175692352029676 2loss/SRqqt2/assignment_loss‚0.6535581372373178 3loss/FBt2/assignment_loss‚0.0011665353740676818 /loss/FRt2/detection_loss‚0.5461340179709815 trainer/global_step‚8349 _runtime‚ 4312.81538095 0loss/FRt2/assignment_loss‚1.7180384598228722 1loss/SRqqt1/detection_loss‚0.5368471994164533 1loss/SRqqt2/detection_loss‚0.8961128189459743 # _timestamp‚1.78449083624074e+09 /loss/FBt2/detection_loss‚1.0578827445528336¥D OÖnï‚(:ã ÛÖôÒ ÚÂË( proc.memory.availableMB‚ 492.35546875# proc.memory.rssMB‚ 2847.69921875* proc.memory.percent‚17.380976676940918$ disk./.usageGB‚1176.505012512207 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚11553.71728515625$ memory_percent‚96.99490070343018 proc.cpu.threads‚17 network.sent‚ 1.1390752e+07 network.recv‚ 1.176343e+06) disk./.usagePercent‚76.63279677656453 disk.nvme0c0n1.in‚ 4.3359375( disk.nvme0c0n1.out‚38820.10205078125& disk.nvme0n1.out‚38820.10205078125( disk.nvme1c1n1.out‚11553.71728515625 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 113.078125" disk.dm-0.out‚43546.2998046875 cpu‚26.087028642577902 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0! disk.dm-1.out‚6584.3916015625 disk.rbd0.out‚ 0.4697265625 disk.nvme0n1.in‚ 4.3359375O":œð‚(:‘ ÛÖôÒúó•Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚52.0 gpu.0.powerWatts‚118.279' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.42633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š.6hñ‚(:Ý êÖôÒÇÊùË" disk.dm-0.out‚43681.6357421875 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0# memory_percent‚96.9965934753418& proc.memory.availableMB‚ 492.078125 proc.cpu.threads‚17 disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚38908.01416015625 network.sent‚ 1.1455016e+07$ disk./.usageGB‚1176.742805480957 disk.dm-0.in‚ 4.3359375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375% disk.nvme1n1.out‚11602.2451171875 network.recv‚ 1.180652e+06 disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625' disk.nvme1c1n1.out‚11602.2451171875 disk.md0.out‚ 113.53125# proc.memory.rssMB‚ 2847.69921875* proc.memory.percent‚17.380976676940918 cpu‚26.08580323905865) disk./.usagePercent‚76.64828565256194" disk.dm-1.out‚6584.53076171875( disk.nvme0c0n1.out‚38908.01416015625 disk.md0.in‚0\Zy’ò‚(:‡ êÖôÒˆôÕË gpu.0.gpu‚46.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚255.348' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚85.116 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï`Ô§2ó- _wandbruntime‚4336  _runtime‚4336™zæÈô ' lr-AdamW/pg1‚0.00019376716686993553 ' lr-AdamW/pg2‚0.00019376716686993553 trainer/global_step‚8399 % _timestamp‚1.7844908603872044e+09 _runtime‚ 4336.96221998 _step‚678œ}ÍÍõÇ ' lr-AdamW/pg1‚0.00019376716686993553 ' lr-AdamW/pg2‚0.00019376716686993553 trainer/global_step‚8399 % _timestamp‚1.7844908603872044e+09 _runtime‚ 4336.96221998 _step‚678¦sùx2ö- _wandbruntime‚4336  _runtime‚4336G:ùšë÷âO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8062328152064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚4336.963081032 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84903903 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚642700754860544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 150326182 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22210418 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8061987365888 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8309932357120 trainer/global_step‚8399 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65432422 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚635967667490960 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚644015512043682 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚635967684530320 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8047827513362 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65432422 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84893760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22210137 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚642700772379136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 _step‚679 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚510523958309376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65432424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44546270 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84893760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚510523933884928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.7<¹ýfreed‚650762742226432 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚650762742226432 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66756690 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44546272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚518834015448064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 $ _timestamp‚1.784490860388211e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84903903 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65432424 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206016 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚642700772379136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚650763100531200 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65432422 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84893760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22210137 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚642700772379136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 _step‚679 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚510523958309376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65432424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44546270 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84893760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚510523933884928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚650762742226432 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚650762742226432 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66756690 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44546272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚518834015448064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 $ _timestamp‚1.784490860388211e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84903903 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65432424 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176§F^’Ñ2ù- _wandbruntime‚4337  _runtime‚4337\ï®ó&úíM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66764027 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8310892981248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 % _timestamp‚1.7844908607478707e+09 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150353718 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8063258070528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚642774946629632 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84913792 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚642774964148224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84903635 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚510582834641920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚510582810217472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375«±r‘ÎûòM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84903635 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚518893852397568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚680 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚642774946629632 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84913792 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65439926 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44551351 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8062917277184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44551353 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84913792 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚642774964148224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84903635 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚510582834641920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚510582810217472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66764027 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8310892981248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 % _timestamp‚1.7844908607478707e+09 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150353718 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8063258070528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚642774946629632 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67138339 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8094270578370 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8108515187712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65807522 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8108855973888 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚646410343314944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85389362 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚639638988654320 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65807524 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚654518858502656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚647732902270008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8357960256000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚646410360833536 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚513467774936576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚521825862071296 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 _runtime‚4361.513893516 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8108515187712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22337740 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8093930655048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44800320 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚654519216807424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85399505 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚654518858502656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65807524 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151196884 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚654518858502656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65807524 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151196884 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67138339 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8094270578370 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8108515187712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65807522 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8108855973888 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚646410343314944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85389362 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚639638988654320 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65807524 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚654518858502656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚647732902270008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8357960256000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚646410360833536 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚513467774936576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚521825862071296 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 _runtime‚4361.513893516 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8108515187712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22337740 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8093930655048«c?82‡- _wandbruntime‚4361  _runtime‚4361Ûщ2ˆíM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 _runtime‚4361.874192518 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚646484552602624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85409394 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚646484535084032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8094858878284 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85399237 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚684 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67145715 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚521885697644544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚513526626844672 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚639712397697440 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚654594338494976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8109445099008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22340593 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚513526651269120 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65815026 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65815024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚654594338494976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚646484552602624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44805401 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8358921601024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85399237 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151224420 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44805403 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚647807256575724 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151224420 J=DeviceSŽlöÁ tatsMonitor.on_train_batch_end/active.large_pool.freed‚65815024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8109785892352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65815026 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22340314 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8095198801718 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151214261 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67145996 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67145996 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 _runtime‚4361.874192518 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚646484552602624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85409394 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚646484535084032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8094858878284 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85399237 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚684 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67145715 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚521885697644544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚513526626844672 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚639712397697440 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚654594338494976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8109445099008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22340593 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚513526651269120 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65815026 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65815024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚654594338494976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚646484552602624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44805401 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8358921601024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85399237 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151224420 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44805403 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚647807256575724 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151224420 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65815024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8109785892352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65815026 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22340314 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8095198801718 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151214261 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚658274974778880 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159691776 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚650119949287936 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚516411591563776 trainer/global_step‚8499 _step‚687 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚658275333083648 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚650119931769344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8155383795712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 152077731 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚658275333083648 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8155043009536 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67519584 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚516411567139328 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚524817695449600 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚651451006421698 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚650119931769344 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8140373720056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚643310275738960 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66182624 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85884964 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85895107 _runtime‚4385.758430733 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚658274974778880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8140713643378 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85895107 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66182622 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22465214 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45054370 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚650119949287936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45054370 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚650119949287936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚658274974778880 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159691776 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚650119949287936 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚516411591563776 trainer/global_step‚8499 _step‚687 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚658275333083648 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚650119931769344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8155383795712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 152077731 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚658275333083648 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8155043009536 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67519584 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚516411567139328 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚524817695449600 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚651451006421698 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚68R—¨=50119931769344 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8140373720056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚643310275738960 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66182624 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85884964 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85895107 _runtime‚4385.758430733 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚658274974778880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8140713643378 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85895107 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66182622 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22465214 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 _runtime‚4386.119143583 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚516470467896320 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚643383701821440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152095122 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85894839 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22468051 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67527211 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8141301943292 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚516470443471872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152095122 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8155972920832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚651525360727526 $ _timestamp‚1.784490909543978e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85904996 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚524877530428928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚651525003764732 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67527504 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0›}"÷&™ñM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85904996 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚524877530428928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚651525003764732 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67527504 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66190124 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 _runtime‚4386.119143583 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚516470467896320 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚643383701821440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152095122 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85894839 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22468051 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67527211 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8141301943292 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚516470443471872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152095122 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8155972920832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚651525360727526 $ _timestamp‚1.784490909543978e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0°ìY ì2š- _wandbruntime‚4386  _runtime‚43862Ÿ¦2›ÿ _runtime‚4386.119803374 0loss/FRt1/assignment_loss‚0.7435980836550394 1loss/FBt1/assignment_loss‚0.02801902065289524 /loss/FRt1/detection_loss‚1.3054161012744043 1loss/SRqqt2/detection_loss‚1.0757032794343842 /loss/FBt2/detection_loss‚0.9484361367940437 $ _timestamp‚1.784490909545005e+09 2loss/SRqqt2/assignment_loss‚0.6715897139653537 /loss/FBt1/detection_loss‚0.5835253442548072 _step‚689 /loss/FRt2/detection_loss‚0.5150822788231026 epoch‚10 trainer/global_step‚8499 0loss/FRt2/assignment_loss‚1.8241043534268435 2loss/SRqqt1/assignment_loss‚0.5275818295694662 0loss/FBt2/assignment_loss‚0.0405073587866267 0loss/SRqqt1/detection_loss‚0.578951816351238 %loss/total_loss‚8.842515316988205Eå œ„ 1loss/SRqqt2/detection_loss‚1.0757032794343842 /loss/FBt2/detection_loss‚0.9484361367940437 $ _timestamp‚1.784490909545005e+09 2loss/SRqqt2/assignment_loss‚0.6715897139653537 /loss/FBt1/detection_loss‚0.5835253442548072 _step‚689 /loss/FRt2/detection_loss‚0.5150822788231026 epoch‚10 trainer/global_step‚8499 0loss/FRt2/assignment_loss‚1.8241043534268435 2loss/SRqqt1/assignment_loss‚0.5275818295694662 0loss/FBt2/assignment_loss‚0.0405073587866267 0loss/SRqqt1/detection_loss‚0.578951816351238 %loss/total_loss‚8.842515316988205 _runtime‚4386.119803374 0loss/FRt1/assignment_loss‚0.7435980836550394 1loss/FBt1/assignment_loss‚0.02801902065289524 /loss/FRt1/detection_loss‚1.3054161012744043±×è¼kn‚(:ã ¦×ô҇ͷË) disk./.usagePercent‚76.57081841296588( disk.nvme0c0n1.out‚38982.12744140625 disk.nvme0n1.in‚ 4.3359375( disk.nvme1c1n1.out‚11661.40673828125 disk.md0.out‚ 115.39453125 disk.dm-0.in‚ 4.3359375$ memory_percent‚96.99594974517822( proc.memory.availableMB‚ 492.18359375 disk.rbd0.in‚0 disk.md0.in‚0# proc.memory.rssMB‚ 2847.69921875 cpu‚26.081466575073843 proc.cpu.threads‚17 network.sent‚ 1.1587737e+07 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚11661.40673828125 disk.dm-1.out‚6585.244140625* proc.memory.percent‚17.380976676940918 network.recv‚ 1.195985e+06% disk./.usageGB‚1175.5534896850586 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375& disk.nvme0n1.out‚38982.12744140625 disk.nvme1n1.in‚0 disk.dm-0.out‚43810.21484375 disk.dm-1.in‚0åá㜞‚(:‘ ¦×ôÒ¾þ©Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚118.376' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.45866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]„ÏžŸ‚(:“ µ×ôÒ­ñúË gpu.0.gpu‚81.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚253.949' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.64966666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m“DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚655168396648024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8186816785064 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 152938288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 cQDeviceStatsMonitor.on_train_batch_sta \|éþrt/inactive_split_bytes.small_pool.allocated‚ 8454044140544 _step‚691 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚519355408190976 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚662031091055104 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚662031091055104 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚646981596902320 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86380566 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚653829537742336 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66557724 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚527809452331520 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚527809294736896 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66557722 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8201570831360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 _runtime‚4410.473552077 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67901016¯´o î'¦èO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8201570831360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86380566 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8201911617536 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8187156708386 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66557724 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22592315 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚519355383766528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45308422 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 % _timestamp‚1.7844909338975778e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚653829537742336 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8201911617536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 152948433 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚662031091055104 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚646981596902320 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86380566 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚653829537742336 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66557724 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚527809452331520 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚527809294736896 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66557722 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8201570831360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 _runtime‚4410.473552077 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67901016 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚653829520223744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22592594 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚662031449359872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚655168753610706 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 trainer/global_step‚8549 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67900735 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚653829520223744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚655168396648024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8186816785064 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 152938288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8454044140544 _step‚691 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚519355408190976 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚662031091055104 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8188084931734 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8202500742656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86390441 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8202500742656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45313501 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8454865616384 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 trainer/global_step‚8549 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8187745008300 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66565224 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67908611 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚519414260099072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚655242750953740 _runtime‚4410.833364698 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚653903711992832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 152955665 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22594836 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚662106212735488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8454998779392 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67908337 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66565226 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22595108 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 _step‚692 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86390441 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚662106571047424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45313503 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86400598 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86400598 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152965824 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚519414284523520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚655243107916534 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8202841536000 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152965824 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592¸üøÌ©òM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66565224 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67908611 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚519414260099072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚655242750953740 _runtime‚4410.833364698 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚653903711992832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 152955665 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22594836 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚662106212735488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8454998779392 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67908337 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66565226 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22595108 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 _step‚692 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86390441 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚662106571047424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45313503 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86400598 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86400598 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152965824 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚519414284523520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚655243107916534 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8202841536000 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152965824 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8188084931734 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8202500742656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86390441 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8202500742656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45313501 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8454865616384 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 trainer/global_step‚8549 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8187745008300´®-ø2ª- _wandbruntime‚4410  _runtime‚4410u3«‚ 1loss/FBt1/assignment_loss‚0.02080502929095803 2loss/FBt2/assignment_loss‚0.009247058841638114 /loss/FRt2/detection_loss‚0.4834587535957957 1loss/SRqqt1/detection_loss‚0.5658405049491314 %loss/total_loss‚8.207353862371537 % _timestamp‚1.7844909342591758e+09 2loss/SRqqt2/assignment_loss‚0.5487540545194255 /loss/FBt2/detection_loss‚0.5501690183479625 trainer/global_step‚8549 0loss/FRt1/assignment_loss‚0.8694916013556878 3loss/SRqqt1/assignment_loss‚0.45483093288433885 /loss/FRt1/detection_loss‚1.3574350234656327 0loss/FRt2/assignment_loss‚1.8556590794768792 0loss/SRqqt2/detection_loss‚0.936908008517801 .loss/FBt1/detection_loss‚0.554754797126288 epoch‚10 _runtime‚4410.833864669 _step‚693¨Ã¾ ¬‡ .loss/FBt1/detection_loss‚0.554754797126288 epoch‚10 _runtime‚4410.833864669 _step‚693 1loss/FBt1/assignment_loss‚0.02080502929095803 2loss/FBt2/assignment_loss‚0.009247058841638114 /loss/FRt2/detection_loss‚0.4834587535957957 1loss/SRqqt1/detection_loss‚0.5658405049491314 %loss/total_loss‚8.207353862371537 % _timestamp‚1.7844909342591758e+09 2loss/SRqqt2/assignment_loss‚0.5487540545194255 /loss/FBt2/detection_loss‚0.5501690183479625 trainer/global_step‚8549 0loss/FRt1/assignment_loss‚0.8694916013556878 3loss/SRqqt1/assignment_loss‚0.45483093288433885 /loss/FRt1/detection_loss‚1.3574350234656327 0loss/FRt2/assignment_loss‚1.8556590794768792 0loss/SRqqt2/detection_loss‚0.936908008517801µ‰pºCl­‚(:á Ä×ôÒéÀÄË( disk.nvme0c0n1.out‚39153.93994140625 disk.nvme0n1.in‚ 4.3359375$ disk.nvme1n1.out‚11690.052734375$ memory_percent‚96.99795246124268 proc.cpu.threads‚17 network.sent‚ 1.1653947e+07 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 cpu‚26.079563067665276) disk./.usagePercent‚76.59076019172811% disk./.usageGB‚1175.8596458435059 disk.md0.in‚0" disk.dm-0.out‚44008.7705078125& disk.nvme1c1n1.out‚11690.052734375( proc.memory.availableMB‚ 491.85546875# proc.memory.rssMB‚ 2847.69921875* proc.memory.percent‚17.380976676940918 network.recv‚ 1.202363e+06 disk.md0.out‚ 116.078125& disk.nvme0n1.out‚39153.93994140625 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0" disk.dm-1.out‚6585.63330078125²rE/ž®‚(:“ Ä×ôÒ´¡ïË gpu.0.gpu‚94.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚267.074' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.02466666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]Ä”ª2¯- _wandbruntime‚4435  _runtime‚4435úpJžÅ°¿ & lr-AdamW/pg2‚0.0001927825877492345 trainer/global_step‚8599 # _timestamp‚1.78449095856612e+09 _runtime‚4435.141093916 _step‚694 & lr-AdamW/pg1‚0.0001927825877492345þ$ZN汀 & lr-AdamW/pg1‚0.0001927825877492345 & lr-AdamW/pg2‚0.0001927825877492345 trainer/global_step‚8599 # _timestamp‚1.78449095856612e+09 _runtime‚4435.141093916 _step‚694¶¢ìgL2²- _wandbruntime‚4435  _runtime‚4435é|“é'³ãO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68282550 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22720078 % _timestamp‚1.7844909585669909e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚522299224818176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8248439439360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86886311 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚530801277505536 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66932822 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚530801130396672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚665787565636096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8248439439360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _step‚695 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66932822 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45562470 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22719792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147108864 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122684416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86876168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86876168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚665787207331328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚657539126196736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚665787565636096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66932824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153819135 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚650652901026320 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 153808990 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260gF‹Ôî'´èO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _step‚695 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66932822 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45562470 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22719792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147108864 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122684416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86876168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86876168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚665787207331328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚657539126196736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚665787565636096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66932824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153819135 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚650652901026320 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 153808990 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68282550 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22720078 % _timestamp‚1.7844909585669909e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚522299224818176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8248439439360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86886311 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚530801277505536 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66932822 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚530801130396672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚665787565636096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8248439439360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520·Y- V2µ- _wandbruntime‚4435  _runtime‚4435|ü»%¶íM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚658960498142748 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153836526 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86886043 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153826367 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8249369357824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8234188073308 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚665862329011712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 % _timestamp‚1.7844909589296947e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8249369357824 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68290157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚657613317965824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153836526 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚650726327108800 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚650726310069440 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚522358101150720 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 H@DeviceStatsMonitor.on_tr:RÎain_batch_end/segment.large_pool.current‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153826367 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 trainer/global_step‚8599 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22722322 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚530861112679424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚4435.504908111 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66940326 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8234527996742 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚665862687323648 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚657613300447232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22722604 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45567553 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 _step‚696 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66940324 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68289873 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434¾3ibø&·òM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 trainer/global_step‚8599 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22722322 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚530861112679424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚4435.504908111 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66940326 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8234527996742 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚665862687323648 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚657613300447232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22722604 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45567553 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 _step‚696 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66940324 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68289873 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚658960498142748 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153836526 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86886043 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153826367 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8249369357824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8234188073308 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚665862329011712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 % _timestamp‚1.7844909589296947e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8249369357824 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68290157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚657613317965824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153836526 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚650726327108800 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚650726310069440 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚522358101150720 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153826367 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0¸ÛÙA(2¸- _wandbruntime‚4435  _runtime‚4435 ] ¹… 0loss/FRt2/assignment_loss‚1.8552724909451273 2loss/SRqqt1/assignment_loss‚0.5918017328647446 3loss/SRqqt2/assignment_loss‚0.45728952633160386 1loss/SRqqt1/detection_loss‚0.5778881052981479 /loss/FBt2/detection_loss‚0.6094339888256329 epoch‚10 % _timestamp‚1.7844909589306726e+09 0loss/FRt1/assignment_loss‚0.8553431121898549 1loss/SRqqt2/detection_loss‚0.9427543452539895 /loss/FBt1/detection_loss‚0.6488115061416894 trainer/global_step‚8599 3loss/FBt2/assignment_loss‚0.0031721564393410407 _runtime‚4435.505460822 2loss/FBt1/assignment_loss‚0.002351877974639788 /loss/FRt1/detection_loss‚1.4228752403204064 /loss/FRt2/detection_loss‚0.5272073011985614 $loss/total_loss‚8.49420138378374 _step‚697 麊 /loss/FRt2/detection_loss‚0.5272073011985614 $loss/total_loss‚8.49420138378374 _step‚697 0loss/FRt2/assignment_loss‚1.8552724909451273 2loss/SRqqt1/assignment_loss‚0.5918017328647446 3loss/SRqqt2/assignment_loss‚0.45728952633160386 1loss/SRqqt1/detection_loss‚0.5778881052981479 /loss/FBt2/detection_loss‚0.6094339888256329 epoch‚10 % _timestamp‚1.7844909589306726e+09 0loss/FRt1/assignment_loss‚0.8553431121898549 1loss/SRqqt2/detection_loss‚0.9427543452539895 /loss/FBt1/detection_loss‚0.6488115061416894 trainer/global_step‚8599 3loss/FBt2/assignment_loss‚0.0031721564393410407 _runtime‚4435.505460822 2loss/FBt1/assignment_loss‚0.002351877974639788 /loss/FRt1/detection_loss‚1.4228752403204064¹!Œ»@j»‚(:ß Ó×ôÒ³ÞõË# disk.nvme1n1.out‚11737.50390625 disk.md0.out‚ 116.41015625( disk.nvme0c0n1.out‚39178.01220703125& disk.nvme0n1.out‚39178.01220703125# proc.memory.rssMB‚ 2847.69921875 proc.cpu.threads‚17 network.recv‚ 1.206714e+06 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚11737.50390625 disk.nvme1n1.in‚0 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0" disk.dm-1.out‚6586.21923828125 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375* proc.memory.percent‚17.380976676940918 cpu‚26.07861680550818 network.sent‚ 1.1729218e+07% disk./.usageGB‚1173.4862594604492 disk.md0.in‚0" disk.dm-0.out‚44078.9931640625# memory_percent‚96.9996690750122( proc.memory.availableMB‚ 491.57421875) disk./.usagePercent‚76.43616736430218ÔLž¼‚(:“ Ó×ôÒЙéË gpu.0.gpu‚91.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚211.781' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚70.59366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û”k½‚(:à â×ôÒ¨¶Ë disk.md0.in‚0! disk.dm-1.out‚6586.3603515625' proc.memory.availableMB‚ 491.3203125# proc.memory.rssMB‚ 2847.69921875* proc.memory.percent‚17.380976676940918 cpu‚26.07768768651153 network.sent‚ 1.1778799e+07 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375' disk.nvme0c0n1.out‚39236.0361328125% disk.nvme0n1.out‚39236.0361328125' disk.nvme1c1n1.out‚12085.6689453125 proc.cpu.threads‚17 network.recv‚ 1.211008e+06 disk.md0.out‚ 116.75390625 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625$ memory_percent‚97.00121879577637) disk./.usagePercent‚76.43605878115905% disk./.usageGB‚1173.4845924377441" disk.dm-0.out‚44484.2900390625 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0% disk.nvme1n1.out‚12085.6689453125.8`<ž¾‚(:“ â×ôÒ¨ùÇË gpu.0.gpu‚69.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚249.875' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚83.29166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ËÌH2¿- _wandbruntime‚4459  _runtime‚44597PÉÀà % _timestamp‚1.7844909831175952e+09 _runtime‚4459.692596828 _step‚698 ' lr-AdamW/pg1‚0.00019253297506213212 ' lr-AdamW/pg2‚0.00019253297506213212 trainer/global_step‚8649:s“*ÎÁÈ _step‚698 ' lr-AdamW/pg1‚0.00019253297506213212 ' lr-AdamW/pg2‚0.00019253297506213212 trainer/global_step‚8649 % _timestamp‚1.7844909831175952e+09 _runtime‚4459.692596828ºÓ&]K2Â- _wandbruntime‚4459  _runtime‚4459û…ÍÙé'ÃãO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67307922 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚661248714651136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87371770 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8549933127680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8280042838402 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8550055812096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚661248714651136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22847398 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 _step‚699 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚669543681912320 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87371770 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚661248697132544 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154679692 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚669543323607552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚662603891026040 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8279702915080 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚525243041445376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67307924 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚669543681912320 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚661248697132544 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 % _timestamp‚1.7844909831184826e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚533792950148608 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚533793097257472 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚525243017020928 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68663920 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8294626475008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8294967261184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8294967261184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚654324188110960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 _runtime‚4459.693345909 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67307924 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚8649 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚661248714651136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87371770 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8549933127680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8280042838402 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/r._»°_"eserved_bytes.large_pool.peak‚ 13862174720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8550055812096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚661248714651136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22847398 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 _step‚699 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚669543681912320 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87371770 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚661248697132544 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154679692 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚669543323607552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚662603891026040 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8279702915080 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚525243041445376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67307924 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚669543681912320 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚661248697132544 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 % _timestamp‚1.7844909831184826e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚533792950148608 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚533793097257472 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚525243017020928 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68663920 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8294626475008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8294967261184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8294967261184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚654324188110960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 _runtime‚4459.693345909 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67307924 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚8649 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935»Xcd€2Å- _wandbruntime‚4460  _runtime‚4460µÇ0ó&ÆíM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚661322888901632 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚525301917777920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚661322906420224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22849920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8551019777536 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154697069 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚661322906420224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚662678245331756 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 _runtime‚4460.057324131 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87381645 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45821601 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚533852937555456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚654397614193440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67315426 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87381645 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8295897179648 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154697069 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8295897179648 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚669618803599872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45821603 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67315424 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8295556386304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68671238 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154707228 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87391802 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚661322888901632 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚669618445287936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8295556386304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22849637 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67315424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8550890808832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚8649 _step‚700 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67315426 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚662678602294550 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 % _timestamp‚1.7844909834821367e+09 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154707228 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚669618803599872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚533852784162304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0ž²Y¿ø&ÇòM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚654397631232800 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚525301893353472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8280971061750 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8280631138316 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154707228 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚669618803599872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚533852784162304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚661322888901632 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚525301917777920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚661322906420224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22849920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8551019777536 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154697069 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚661322906420224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚662678245331756 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 _runtime‚4460.057324131 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87381645 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45821601 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚533852937555456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚654397614193440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67315426 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87381645 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8295897179648 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154697069 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8295897179648 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚669618803599872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45821603 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67315424 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8295556386304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68671238 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154707228 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87391802 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚661322888901632 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚669618445287936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8295556386304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22849637 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67315424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8550890808832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚8649 _step‚700 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67315426 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚662678602294550 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 % _timestamp‚1.7844909834821367e+09 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46070572 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚528186833648128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497472 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚664958303105536 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46070570 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67683024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8326145980088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚673299798188544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22974465 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8598017252352 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚666321638215048 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚536784719827456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8326485903410 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69045330 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚528186858072576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚673299439883776 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ƒcqì'ÒæO trainer/global_step‚8699 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87877515 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8597886179328 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚664958285586944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚664958285586944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 155550394 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87877515 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8341495083008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚673299439883776 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784491007374556e+09 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚657995509274320 _step‚703 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 155560539 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚666321995177730 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67683024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 4483.94946267 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67683022 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚673299439883776 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8341495083008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8341154296832 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚536784875324928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46070572 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚528186833648128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497472 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚664958303105536 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46070570 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67683024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8326145980088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚673299798188544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22974465 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8598017252352 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚666321638215048 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚536784719827456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8326485903410 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69045330 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚528186858072576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚673374919876096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 155577930 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67690524 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67690524 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8327074203324 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69052595 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87877247 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8342084208128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚665032494874624 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚536844555074048 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8342084208128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚666395992520764 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚665032494874624 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87877247 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46075653 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚4484.311532911 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69052872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67690526 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚673374561564160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8342425001472 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚528245709980672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 % _timestamp‚1.7844910077363782e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 155567771 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚528245734405120 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚666396349483558 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚665032477356032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87887404 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155577930 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8598845093376‘~ýQÉ ÕòM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 155567771 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚528245734405120 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚666396349483558 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚665032477356032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87887404 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155577930 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8598845093376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 155567771 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22976944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚673374919876096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 155577930 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67690524 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67690524 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8327074203324 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69052595 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87877247 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8342084208128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚665032494874624 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚536844555074048 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8342084208128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚666395992520764 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚665032494874624 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87877247 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46075653 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚4484.311532911 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69052872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67690526 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚673374561564160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8342425001472 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚528245709980672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 % _timestamp‚1.7844910077363782e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675À/ÆŽV2Ö- _wandbruntime‚4484  _runtime‚4484f—ò½×‚ 2loss/SRqqt1/assignment_loss‚0.6168214929139294 2loss/FBt1/assignment_loss‚0.017008536220744257 1loss/SRqqt1/detection_loss‚0.6079141911037667 2loss/SRqqt2/assignment_loss‚0.5432294567255304 /loss/FRt2/detection_loss‚0.5440744393424265 trainer/global_step‚8699 % _timestamp‚1.7844910077373736e+09 /loss/FRt1/assignment_loss‚0.945959112374112 $loss/total_loss‚9.01845684684747 epoch‚10 _runtime‚4484.312036841 _step‚705 0loss/FRt2/assignment_loss‚1.9108211383098315 3loss/FBt2/assignment_loss‚0.0012584604168068836 .loss/FRt1/detection_loss‚1.553275463043974 1loss/SRqqt2/detection_loss‚0.9267925439340373 /loss/FBt1/detection_loss‚0.4433660116476508 /loss/FBt2/detection_loss‚0.9079360008146614éŸ@) ؇ trainer/global_step‚8699 % _timestamp‚1.7844910077373736e+09 /loss/FRt1/assignment_loss‚0.945959112374112 $loss/total_loss‚9.01845684684747 epoch‚10 _runtime‚4484.312036841 _step‚705 0loss/FRt2/assignment_loss‚1.9108211383098315 3loss/FBt2/assignment_loss‚0.0012584604168068836 .loss/FRt1/detection_loss‚1.553275463043974 1loss/SRqqt2/detection_loss‚0.9267925439340373 /loss/FBt1/detection_loss‚0.4433660116476508 /loss/FBt2/detection_loss‚0.9079360008146614 2loss/SRqqt1/assignment_loss‚0.6168214929139294 2loss/FBt1/assignment_loss‚0.017008536220744257 1loss/SRqqt1/detection_loss‚0.6079141911037667 2loss/SRqqt2/assignment_loss‚0.5432294567255304 /loss/FRt2/detection_loss‚0.5440744393424265ÁAXêËpÙ‚(:å €ØôÒÑšÞË disk.rbd0.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 117.48828125 disk.dm-0.in‚ 4.3359375" disk.dm-0.out‚44493.5263671875$ memory_percent‚97.00007438659668# proc.memory.rssMB‚ 2847.69921875) disk./.usagePercent‚76.43573079546351 disk.rbd0.out‚ 0.4697265625 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚12088.47314453125! disk.dm-1.out‚6586.5849609375' proc.memory.availableMB‚ 491.5078125 cpu‚26.075554384752742 network.sent‚ 1.1846083e+07% disk./.usageGB‚1173.4795570373535( disk.nvme0c0n1.out‚39244.28955078125& disk.nvme0n1.out‚39244.28955078125 disk.dm-1.in‚0* proc.memory.percent‚17.380976676940918 proc.cpu.threads‚17 network.recv‚ 1.218883e+06 disk.nvme0c0n1.in‚ 4.3359375& disk.nvme1n1.out‚12088.47314453125žîóý“Ú‚(:ˆ €ØôÒèäÄÌ gpu.0.gpu‚50.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚249.678' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚83.226 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mB½mÛ‚(:â ØôÒŠÿ‘Ì cpu‚26.07463386204836 disk.md0.out‚ 117.8359375( proc.memory.availableMB‚ 491.23046875* proc.memory.percent‚17.380976676940918 proc.cpu.threads‚17 disk.rbd0.out‚ 0.4697265625 network.sent‚ 1.1910281e+07% disk./.usageGB‚1173.4795188903809 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0! disk.dm-1.out‚6586.6904296875( disk.nvme0c0n1.out‚39246.40380859375 network.recv‚ 1.222936e+06) disk./.usagePercent‚76.43572831072339 disk.nvme1n1.in‚0% disk.nvme1n1.out‚12089.1123046875 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚12089.1123046875# memory_percent‚97.0017671585083# disk.dm-0.out‚44495.42724609375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375& disk.nvme0n1.out‚39246.40380859375# proc.memory.rssMB‚ 2847.69921875¤6Èn“Ü‚(:ˆ ØôÒ¦ìƒÌ gpu.0.gpu‚91.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚265.899' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚88.633 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ±œ2Ý- _wandbruntime‚4508  _runtime‚4508eb4@ÈÞ % _timestamp‚1.7844910319113846e+09 _runtime‚ 4508.48638617 _step‚706 ' lr-AdamW/pg1‚0.00019202961610827155 ' lr-AdamW/pg2‚0.00019202961610827155 trainer/global_step‚8749ç/ÜÍßÇ _runtime‚ 4508.48638617 _step‚706 ' lr-AdamW/pg1‚0.00019202961610827155 ' lr-AdamW/pg2‚0.00019202961610827155 trainer/global_step‚8749 % _timestamp‚1.7844910319113846e+09ÂH¶L2à- _wandbruntime‚4508  _runtime‚4508N¥cè'áâO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88362974 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68058124 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚531130674699776 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚539776518994944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8372589045096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚539776676589568 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68058124 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚8749 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68058122 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8387682118656 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156431241 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚677055556160000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46324620 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8372928968418 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46324622 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156421096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚668667891559936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚668667891559936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚677055556160000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68058122 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88362974 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚670039742366738 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 $ _timestamp‚1.784491031912288e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚531130650275328 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 _runtime‚4508.487115872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚668667874041344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23102151 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚670039385404056 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88373117 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚677055914464768 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚677055914464768 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23102437 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8646001889792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88373117 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8645868719616 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8388022904832 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69427059 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚539776676589568 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68058124 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚8749 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68058122 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8387682118656 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156431241 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚677055556160000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46324620 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8372928968418 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46324622 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156421096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚668667891559936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚668667891559936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚677055556160000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68058122 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88362974 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚670039742366738 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 $ _timestamp‚1.784491031912288e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚531130650275328 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240Ã<ÁÕÚ2ã- _wandbruntime‚4508  _runtime‚4508égLäìM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156438473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 K>DeviceStatsMonitor.on_trainõHkRÛ%_batch_end/inactive_split.all.freed‚69434368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156448632 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159684608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚531189551032320 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚531189526607872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚668742083329024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88372849 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8388952823296 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784491032269528e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8646821792768 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88383006 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68065624 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚668742065810432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚668742065810432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8646957052928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚677130677840384 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23104667 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚668742083329024 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135260160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚670114096672566 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68065626 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8373857191766 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69434655 trainer/global_step‚8749 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚539836508085248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚661740222441440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚661740239480800ÔŽž´÷&åñM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156448632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8388612029952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23104952 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8388952823296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156448632 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159684608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚531189551032320 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚531189526607872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚668742083329024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88372849 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8388952823296 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784491032269528e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8646821792768 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88383006 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68065624 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚668742065810432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚668742065810432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8646957052928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚677130677840384 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23104667 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚668742083329024 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135260160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚670114096672566 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68065626 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8373857191766 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69434655 trainer/global_step‚8749 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚539836508085248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚661740222441440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚661740239480800 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156438473 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69434368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚672377462495744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46578670 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69808197 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8434550726656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23229813 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88858576 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚542768447610368 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 trainer/global_step‚8799 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68433222 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.784491057059952e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68433222 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚534074466902528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8693831501824 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8419032110104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68433224 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚665338100482960 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚680811672436224 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚542768298404352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8434550726656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46578672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚672377462495744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 JBDeviceStatsMonito-}2 er.on_train_batch_start/segment.small_pool.current‚260 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚672377480014336 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8434209940480 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚672377480014336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 157291798 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68433222 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.784491057059952e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68433222 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚534074466902528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8693831501824 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8419032110104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68433224 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚665338100482960 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚680811672436224 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚542768298404352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8434550726656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46578672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚672377462495744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚672377480014336 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8434209940480 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚672377480014336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 157291798 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚672377462495744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46578670 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69808197 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8434550726656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23229813 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88858576 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚542768447610368 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 trainer/global_step‚8799Ç¡½ÙÈ2ó- _wandbruntime‚4534  _runtime‚4534ÞZà¿ó&ôíM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68440726 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚542828131654144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69816145 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69815850 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 % _timestamp‚1.7844910574374106e+09 _runtime‚4534.012533215 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 157309175 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚680887152428544 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚672451671783424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚673831843861574 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8420300256774 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚534133367659520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23232099 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88868451 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46583753 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 _step‚712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8694788419072 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚295 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68440724 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68440724 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚665411526565440 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8419960333340 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚672451671783424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 trainer/global_step‚8799 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚672451654264832 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8435480645120 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 157309175 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 157319334 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88868451 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚665411543604800 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚680887152428544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157319334 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚534133343235072 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚680886794116608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚673831486898780 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8435480645120 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23232392 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88878608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0˜]¡qø&õòM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8435480645120 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23232392 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88878608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68440726 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚542828131654144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69816145 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69815850 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 % _timestamp‚1.7844910574374106e+09 _runtime‚4534.012533215 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 157309175 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚680887152428544 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚672451671783424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚673831843861574 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8420300256774 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚534133367659520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23232099 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88868451 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46583753 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 _step‚712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8694788419072 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚295 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68440724 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68440724 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚665411526565440 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8419960333340 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚672451671783424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 trainer/global_step‚8799 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚672451654264832 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8435480645120 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 157309175 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 157319334 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88868451 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚665411543604800 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚680887152428544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157319334 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚534133343235072 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚680886794116608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚673831486898780 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520È=$r2ö- _wandbruntime‚4534  _runtime‚4534†šQ.÷ƒ 3loss/FBt2/assignment_loss‚0.0023545325749731475 %loss/total_loss‚8.125657238102615 epoch‚10 trainer/global_step‚8799 2loss/SRqqt2/assignment_loss‚0.5841251236047053 1loss/SRqqt1/detection_loss‚0.5689079933929876 /loss/FBt1/detection_loss‚0.3308223775883734 /loss/FBt2/detection_loss‚0.5066997109069711 .loss/FRt2/detection_loss‚0.565787223754814 0loss/FRt1/assignment_loss‚0.7982001393294531 2loss/SRqqt1/assignment_loss‚0.5706831660591791 1loss/FBt1/assignment_loss‚0.08584333457686628 /loss/FRt1/îŽÊƒÛdetection_loss‚1.4515110330794714 1loss/SRqqt2/detection_loss‚0.9507458140814884 % _timestamp‚1.7844910574385664e+09 _runtime‚4534.013327467 _step‚713 0loss/FRt2/assignment_loss‚1.7099767891533322]N|øˆ 2loss/SRqqt2/assignment_loss‚0.5841251236047053 1loss/SRqqt1/detection_loss‚0.5689079933929876 /loss/FBt1/detection_loss‚0.3308223775883734 /loss/FBt2/detection_loss‚0.5066997109069711 .loss/FRt2/detection_loss‚0.565787223754814 0loss/FRt1/assignment_loss‚0.7982001393294531 2loss/SRqqt1/assignment_loss‚0.5706831660591791 1loss/FBt1/assignment_loss‚0.08584333457686628 /loss/FRt1/detection_loss‚1.4515110330794714 1loss/SRqqt2/detection_loss‚0.9507458140814884 % _timestamp‚1.7844910574385664e+09 _runtime‚4534.013327467 _step‚713 0loss/FRt2/assignment_loss‚1.7099767891533322 3loss/FBt2/assignment_loss‚0.0023545325749731475 %loss/total_loss‚8.125657238102615 epoch‚10 trainer/global_step‚8799Éï«Hnnù‚(:ã ¼ØôÒˆÿˆÌ network.sent‚ 1.1993398e+07 disk.rbd0.out‚ 0.4697265625 disk.nvme1c1n1.in‚0( proc.memory.availableMB‚ 491.11328125# proc.memory.rssMB‚ 2847.69921875 cpu‚26.077133346009525 proc.cpu.threads‚17 network.recv‚ 1.233841e+06) disk./.usagePercent‚76.61561554407301 disk.nvme1n1.in‚0& disk.nvme1n1.out‚12091.35595703125 disk.md0.in‚0" disk.dm-0.out‚45641.4970703125 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 4.3359375% disk./.usageGB‚1176.2412376403809( disk.nvme0c0n1.out‚40393.27099609375 disk.nvme0n1.in‚ 4.3359375( disk.nvme1c1n1.out‚12091.35595703125 disk.md0.out‚ 119.08984375 disk.dm-0.in‚ 4.3359375 disk.dm-1.out‚ 6587.07421875 disk.rbd0.in‚0& disk.nvme0n1.out‚40393.27099609375# memory_percent‚97.0024824142456* proc.memory.percent‚17.380976676940918A¶Xœú‚(:’ ¼ØôÒ¯·ÁÌ gpu.0.gpu‚22.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚203.642' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.88066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>Œ×iû‚(:Þ ËØôÒÙÖáË disk.dm-1.out‚ 6587.109375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 4.3359375 disk.dm-0.in‚ 4.3359375 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375$ disk.nvme0n1.out‚41814.806640625 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚12311.15771484375 disk.nvme1n1.in‚0& disk.nvme1n1.out‚12311.15771484375 network.recv‚ 1.238002e+06# disk.dm-0.out‚47282.34423828125( proc.memory.availableMB‚ 489.05078125* proc.memory.percent‚17.392516136169434) disk./.usagePercent‚76.61262118375723 disk.md0.in‚0$ memory_percent‚97.01507091522217# proc.memory.rssMB‚ 2849.58984375 cpu‚26.091040437956348 proc.cpu.threads‚17& disk.nvme0c0n1.out‚41814.806640625 network.sent‚ 1.204494e+07% disk./.usageGB‚1176.1952667236328 disk.md0.out‚ 119.29296875 disk.dm-1.in‚0ñþ›ü‚(: ËØôÒ夘Ì gpu.0.gpu‚35.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚266.6' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.86666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wbØE2ý- _wandbruntime‚4560  _runtime‚4560J´´,=þÿy ("Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/*FRt1SRqqt2FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_1_of_5‚1 _runtime‚4560.477080504 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740740299224854 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.224736288189888 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.038518600165843964 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_4_of_5‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3073960244655609 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2398882359266281 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9916194677352905 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.394985556602478 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07075683027505875 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6879198551177979 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0325649194419384 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6679040193557739 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7022965550422668 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8469319343566895 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9965828657150269 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.538794994354248 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15008243918418884 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_5_of_5‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6213738918304443 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28562667965888977 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7308335304260254 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23266051709651947 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6289019584655762 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03672946244478226 )loss/val_total_loss‚8.201368650923204 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.24770395457744598 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9885439872741699 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.995559573173523 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7258694171905518 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6497982740402222 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8681414723396301 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2000945657491684 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7289043068885803 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.790929913520813 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2777777910232544 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9942002296447754 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8132832646369934 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9900647401809692 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5517317652702332 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4204545319080353 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9243058562278748 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6211220622062683 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5177083015441895 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9905093312263489 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7099999785423279 +jet/accuracy_3_of_4‚0.09073010832071304 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6170982122421265 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9954690337181091 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.47904568910598755 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2579024136066437 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_4‚1 jet/accuracy_2_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_3_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7198993563652039 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6144475340843201 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9962999224662781 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6539347767829895 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7964127659797668 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9101777672767639 particle/accuracy_1_of_5‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02786753885447979 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6487124562263489 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 jet/accuracy_5_of_5‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11562500149011612 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.721794843673706 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21357354521751404 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2479759007692337 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7825201749801636 % _timestamp‚1.7844910839015183e+09 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7416929602622986 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.25885719060897827 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2590620517730713 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22368839383125305 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7064834833145142 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1386975198984146 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7854856848716736 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7650104761123657 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7389454245567322 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05566234886646271 +particle/sensitivity‚0.6352782249450684 jet/accuracy_4_of_5‚0 _step‚714 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9965674877166748 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 particle/accuracy_1_of_2‚1 7validation_average_jet_accuracy‚0.38248810172080994 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8166666626930237 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3391246795654297 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980530738830566 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9983673095703125 .particle/accuracy_4_of_4‚0.957861602306366 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9967948794364929 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8960692882537842 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16427083313465118 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18990451097488403 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4273431897163391 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30387255549430847 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.031281329691410065 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5411936640739441 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9770880937576294 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6850479245185852 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.05775900557637215 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03361364081501961 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6904358863830566 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6727220416069031 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9991496801376343 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_3_of_5‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9946359992027283 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.713835597038269 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_4‚1 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.786403477191925 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9983673095703125 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.724826991558075 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2155301719903946 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 )jet/accuracy_1_of_2‚0.790709376335144 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7460368275642395 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9187103509902954 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9945970177650452 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.7984932661056519 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8469319343566895 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45123401284217834 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5982781648635864 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2170097976922989 'particle/f_score‚0.6469810009002686 & Purity/0FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6597407460212708 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.034999698400497437 +jet/accuracy_3_of_3‚0.07157355546951294 ,jet/accuracy_4_of_4‚0.016514042392373085 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.25934410095214844 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7718873023986816 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5798611044883728 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6597407460212708 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.23988378047943115 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8685564398765564 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24695345759391785 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970942735671997 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5360878705978394 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988153576850891 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5730231404304504 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4699999988079071 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6562248468399048 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.16758543252944946 particle/accuracy_2_of_3‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7001211643218994 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 )jet/accuracy_2_of_3‚0.589911699295044 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 epoch‚10 +jet/accuracy_1_of_1‚0.34069734811782837 particle/accuracy_2_of_5‚1 trainer/global_step‚8810 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9300217628479004 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04529617726802826 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2479759007692337 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21205919981002808 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45043328404426575 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15246149897575378 particle/accuracy_2_of_2‚1 (particle/accuracy‚0.8511477112770081 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.24240367114543915 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9967328310012817 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1997634619474411 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7233539819717407 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7099999785423279 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_1‚1 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02786753885447979 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8039473295211792 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3927385210990906 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 *jet/accuracy_1_of_4‚0.9700671434402466 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8140110373497009ô;’4ÿ„z 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.21205919981002808 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45043328404426575 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15246149897575378 particle/accuracy_2_of_2‚1 (particle/accuracy‚0.8511477112770081 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.24240367114543915 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9967328310012817 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1997634619474411 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7233539819717407 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7099999785423279 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_1‚1 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02786753885447979 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8039473295211792 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3927385210990906 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 *jet/accuracy_1_of_4‚0.9700671434402466 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8140110373497009 ("Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/*FRt1SRqqt2FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_1_of_5‚1 _runtime‚4560.477080504 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740740299224854 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.224736288189888 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.038518600165843964 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_4_of_5‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3073960244655609 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2398882359266281 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9916194677352905 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.394985556602478 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07075683027505875 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6879198551177979 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0325649194419384 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6679040193557739 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7022965550422668 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8469319343566895 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9965828657150269 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.538794994354248 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15008243918418884 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_5_of_5‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6213738918304443 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28562667965888977 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7308335304260254 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23266051709651947 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6289019584655762 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03672946244478226 )loss/val_total_loss‚8.201368650923204 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.24770395457744598 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9885439872741699 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.995559573173523 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7258694171905518 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6497982740402222 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8681414723396301 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2000945657491684 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7289043068885803 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.790929913520813 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2777777910232544 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9942002296447754 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8132832646369934 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9900647401809692 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5517317652702332 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4204545319080353 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9243058562278748 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6211220622062683 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5177083015441895 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9905093312263489 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7099999785423279 +jet/accuracy_3_of_4‚0.09073010832071304 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6170982122421265 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9954690337181091 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.47904568910598755 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2579024136066437 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_4‚1 jet/accuracy_2_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_3_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7198993563652039 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6144475340843201 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9962999224662781 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6539347767829895 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7964127659797668 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9101777672767639 particle/accuracy_1_of_5‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02786753885447979 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6487124562263489 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 jet/accuracy_5_of_5‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11562500149011612 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.721794843673706 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21357354521751404 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2479759007692337 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7825201749801636 % _timestamp‚1.7844910839015183e+09 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7416929602622986 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.25885719060897827 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2590620517730713 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22368839383125305 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7064834833145142 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1386975198984146 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7854856848716736 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7650104761123657 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7389454245567322 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05566234886646271 +particle/sensitivity‚0.6352782249450684 jet/accuracy_4_of_5‚0 _step‚714 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9965674877166748 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 particle/accuracy_1_of_2‚1 7validation_average_jet_accuracy‚0.38248810172080994 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8166666626930237 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3391246795654297 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980530738830566 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9983673095703125 .particle/accuracy_4_of_4‚0.957861602306366 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9967948794364929 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8960692882537842 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16427083313465118 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18990451097488403 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4273431897163391 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30387255549430847 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.031281329691410065 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5411936640739441 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9770880937576294 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6850479245185852 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.05775900557637215 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03361364081501961 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6904358863830566 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6727220416069031 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9991496801376343 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_3_of_5‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9946359992027283 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.713835597038269 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_4‚1 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.786403477191925 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9983673095703125 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.724826991558075 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2155301719903946 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 )jet/accuracy_1_of_2‚0.790709376335144 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7460368275642395 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9187103509902954 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9945970177650452 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.7984932661056519 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8469319343566895 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45123401284217834 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5982781648635864 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2170097976922989 'paôv¶xrticle/f_score‚0.6469810009002686 & Purity/0FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6597407460212708 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.034999698400497437 +jet/accuracy_3_of_3‚0.07157355546951294 ,jet/accuracy_4_of_4‚0.016514042392373085 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.25934410095214844 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7718873023986816 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5798611044883728 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6597407460212708 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.23988378047943115 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8685564398765564 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24695345759391785 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970942735671997 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5360878705978394 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988153576850891 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5730231404304504 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4699999988079071 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6562248468399048 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.16758543252944946 particle/accuracy_2_of_3‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7001211643218994 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 )jet/accuracy_2_of_3‚0.589911699295044 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 epoch‚10 +jet/accuracy_1_of_1‚0.34069734811782837 particle/accuracy_2_of_5‚1 trainer/global_step‚8810 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9300217628479004 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04529617726802826 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2479759007692337ʯ5 ž€‚(:“ ÚØôÒäÓË gpu.0.gpu‚76.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚162.349' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚54.11633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ëó¦i‚(:Þ ÚØôÒÏÙ‹Í cpu‚26.09274277565088 network.recv‚ 1.241586e+06% disk./.usageGB‚1176.1952476501465 disk.nvme0c0n1.in‚ 4.3359375& disk.nvme0n1.out‚41817.59521484375( proc.memory.availableMB‚ 737.74609375) proc.memory.percent‚17.39826202392578 proc.cpu.threads‚17 disk.md0.out‚ 119.62890625 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625( disk.nvme0c0n1.out‚41817.59521484375 disk.nvme0n1.in‚ 4.3359375 proc.memory.rssMB‚ 2850.53125) disk./.usagePercent‚76.61261994138717# disk.dm-0.out‚47284.85595703125! disk.dm-1.out‚6832.5869140625 disk.rbd0.in‚0$ disk.nvme1n1.out‚12557.083984375$ memory_percent‚95.49715518951416 network.sent‚ 1.2047862e+07 disk.md0.in‚0 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚12557.083984375 disk.nvme1n1.in‚0™ÔÖ¬2‚- _wandbruntime‚4582  _runtime‚4582ñïƒ=Ƀà _runtime‚4582.100070827 _step‚715 ' lr-AdamW/pg1‚0.00019152077262590014 ' lr-AdamW/pg2‚0.00019152077262590014 trainer/global_step‚8849 % _timestamp‚1.7844911055250154e+094ââ6Î„È % _timestamp‚1.7844911055250154e+09 _runtime‚4582.100070827 _step‚715 ' lr-AdamW/pg1‚0.00019152077262590014 ' lr-AdamW/pg2‚0.00019152077262590014 trainer/global_step‚8849Ëb¼¶2…- _wandbruntime‚4582  _runtime‚4582‡OV7é'†ãO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚4582.101126448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158577379 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚670926492810160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89517748 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚678034059122176 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚678034041603584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70474239 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚547377921675264 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚679416797912064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚686539295420928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23455325 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8766594459136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89527891 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8505594603520 % _timestamp‚1.7844911055261261e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69049486 _step‚716 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8505594603520 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚686539653725696 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚8849 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23455039 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8505253817344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69049488 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158567234 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚686539295420928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935õø¯Kî'‡èO G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89527891 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8505594603520 % _timestamp‚1.7844911055261261e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69049486 _step‚716 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8505594603520 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚686539653725696 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚8849 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23455039 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8505253817344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69049488 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158567234 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚686539295420928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚4582.101126448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158577379 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚670926492810160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89517748 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚678034059122176 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚678034041603584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70474239 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚547377921675264 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚679416797912064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚686539295420928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23455325 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8766594459136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0ÌàáðÈ2ˆ- _wandbruntime‚4582  _runtime‚4582 Á—Ð0 ‰íM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8506183728640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚686614417101312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47024283 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚679490795255098 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8767691566080 trainer/global_step‚8849 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89537780 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚678108233372672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8491233325252 % _timestamp‚1.7844911059058025e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89527623 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89527623 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69056988 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8490893401818 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚547437593871360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚686614775413248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69056990 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158584611 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚670999901853280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69056988 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 168073216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 143648768 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚678108250891264 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89537780 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚547437761944576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158594770 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚678108233372672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚678108250891264 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70481835 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47024281 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23457554 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMon¿ÂÜÃitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚679491152217892 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚670999918892640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 _runtime‚4582.480995312 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8506524521984 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚686614775413248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8506524521984 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158584611 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720÷²ñÅø&ŠòM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 _step‚717 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚538670045954048 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8506183728640 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚538670070378496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8767547917312 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70481835 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47024281 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23457554 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚679491152217892 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚670999918892640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 _runtime‚4582.480995312 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8506524521984 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚686614775413248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8506524521984 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158584611 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8506183728640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚686614417101312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47024283 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚679490795255098 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8767691566080 trainer/global_step‚8849 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89537780 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚678108233372672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8491233325252 % _timestamp‚1.7844911059058025e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89527623 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89527623 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69056988 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8490893401818 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚547437593871360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚686614775413248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69056990 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158584611 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚670999901853280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69056988 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 168073216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 143648768 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚678108250891264 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89537780 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚547437761944576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158594770 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚678108233372672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚678108250891264 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69424586 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8552122425344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23579959 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161788928 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚674597796934160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8536408243590 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90023493 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚8899 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚681743630057984 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚681743647576576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚681743647576576 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚674597779894800 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 _runtime‚4608.911753154 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90013350 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8552122425344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23579675 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69424588 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8536748166912 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90023493 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69424586 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚550369665422848 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚681743630057984 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚683134545101072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚550369503633920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚541555010673152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8814517385216 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90013350 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8551781639168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47273252 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7844911323368435e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚690295770001920 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚690295411697152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 137364480 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69424588 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚690295770001920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47273250 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8814654749696 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚690295411697152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159448081 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70852925 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968€l®—èO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69424586 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8552122425344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23579959 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161788928 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚674597796934160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8536408243590 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90023493 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚8899 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚681743630057984 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚681743647576576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚681743647576576 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚674597779894800 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 _runtime‚4608.911753154 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90013350 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8552122425344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23579675 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69424588 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8536748166912 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90023493 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69424586 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚550369665422848 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚681743630057984 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚683134545101072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚550369503633920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚541555010673152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8814517385216 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDƒ9åc@eviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90013350 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8551781639168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47273252 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7844911323368435e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚690295770001920 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚690295411697152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 137364480 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69424588 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚690295770001920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47273250 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8814654749696 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚690295411697152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159448081 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70852925 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70853211 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0Ð ïú#2˜- _wandbruntime‚4609  _runtime‚4609ÄCYò&™ìM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69432090 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70860448 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚681817839345664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚541613862581248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47278333 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚681817839345664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8553052343808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159455313 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 trainer/global_step‚8899 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 _runtime‚4609.290989907 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90033382 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚690370891689472 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8552711550464 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8537336466826 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47278331 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90023225 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159465472 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69432088 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚683208899406900 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159465472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8537676390260 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23582399 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8552711550464 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 $ _timestamp‚1.784491132715792e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70860732 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161781760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚690370891689472 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90023225 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚681817821827072 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 137357312 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8815476217344 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚674671205977280 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90033382 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8815613574656 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚683208542444106 _step‚721 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69432090 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23582117 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚541613887005696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚550429500580352 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087ŸÂÒ÷&šñM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23582399 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8552711550464 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 $ _timestamp‚1.784491132715792e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70860732 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161781760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚690370891689472 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90023225 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚681817821827072 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 137357312 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8815476217344 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚674671205977280 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90033382 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8815613574656 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚683208542444106 _step‚721 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69432090 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23582117 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚541613887005696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚550429500580352 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69432090 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70860448 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚681817839345664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚541613862581248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47278333 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚681817839345664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8553052343808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159455313 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 trainer/global_step‚8899 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 _runtime‚4609.290989907 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90033382 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚690370891689472 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8552711550464 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8537336466826 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47278331 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90023225 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159465472 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69432088 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚683208899406900 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159465472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8537676390260 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284Ñ~˜ci2›- _wandbruntime‚4609  _runtime‚4609/(lœ‚ 2loss/SRqqt1/assignment_loss‚0.5213729349034717 epoch‚11 % _timestamp‚1.7844911327169743e+09 _step‚722 /loss/FRt1/assignment_loss‚0.803736802529205 /loss/FRt1/detection_loss‚1.3174506104022095 trainer/global_step‚8899 /loss/FRt2/assignment_loss‚1.908611823530758 1loss/FBt1/assignment_loss‚0.06331040824825829 /loss/FRt2/detection_loss‚0.5998152071945009 0loss/SRqqt1/detection_loss‚0.570718243072689 1loss/SRqqt2/detection_loss‚1.2228421041305872 /loss/FBt2/detection_loss‚0.8162499255655954 2loss/SRqqt2/assignment_loss‚0.7319456794500602 4loss/FBt2/assignment_loss‚0.00013602875845701115 /loss/FBt1/detection_loss‚0.4529830702667823 %loss/total_loss‚9.009172838052574 _runtime‚4609.291635318¢Tcí ‡ 0loss/SRqqt1/detection_loss‚0.570718243072689 1loss/SRqqt2/detection_loss‚1.2228421041305872 /loss/FBt2/detection_loss‚0.8162499255655954 2loss/SRqqt2/assignment_loss‚0.7319456794500602 4loss/FBt2/assignment_loss‚0.00013602875845701115 /loss/FBt1/detection_loss‚0.4529830702667823 %loss/total_loss‚9.009172838052574 _runtime‚4609.291635318 2loss/SRqqt1/assignment_loss‚0.5213729349034717 epoch‚11 % _timestamp‚1.7844911327169743e+09 _step‚722 /loss/FRt1/assignment_loss‚0.803736802529205 /loss/FRt1/detection_loss‚1.3174506104022095 trainer/global_step‚8899 /loss/FRt2/assignment_loss‚1.908611823530758 1loss/FBt1/assignment_loss‚0.06331040824825829 /loss/FRt2/detection_loss‚0.5998152071945009Ò{<jž‚(:ß ‡ÙôÒú±ÆË disk.md0.in‚0" disk.dm-0.out‚47344.6826171875 disk.nvme0c0n1.in‚ 4.3359375 proc.memory.rssMB‚ 2850.53125) proc.memory.percent‚17.39826202392578 proc.cpu.threads‚17& disk.nvme0n1.out‚41876.19189453125 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚12806.447265625( disk.nvme0c0n1.out‚41876.19189453125$ memory_percent‚97.00572490692139 cpu‚26.098496185450447 network.sent‚ 1.2171464e+07 network.recv‚ 1.253141e+06) disk./.usagePercent‚76.61572213942407 disk.nvme1c1n1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625( proc.memory.availableMB‚ 490.58203125% disk./.usageGB‚1176.2428741455078 disk.md0.out‚ 120.73046875 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0" disk.dm-1.out‚7078.33642578125 disk.nvme0n1.in‚ 4.3359375& disk.nvme1c1n1.out‚12806.447265625Õ£œŸ‚(:‘ ‡ÙôÒ³¹ Ì gpu.0.gpu‚7.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚251.603' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚83.86766666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N<ka ‚(:Ö –ÙôÒÞŽØË network.sent‚ 1.223455e+07 disk.rbd0.out‚ 0.4697265625( disk.nvme0c0n1.out‚41877.10595703125 disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚41877.10595703125# disk.nvme1n1.out‚12807.19140625$ proc.memory.availableMB‚490.8125 cpu‚26.100542885924447 network.recv‚ 1.256289e+06( disk./.usagePercent‚76.5344785917972" disk.dm-0.out‚47345.2451171875" disk.dm-1.out‚7078.53369140625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375$ memory_percent‚97.00431823730469 proc.cpu.threads‚17% disk./.usageGB‚1174.9955825805664 disk.md0.in‚0 disk.md0.out‚ 121.1484375 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0 proc.memory.rssMB‚ 2850.53125 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚12807.19140625 disk.nvme1n1.in‚0) proc.memory.percent‚17.39826202392578ž¡ nŸ¡‚(:” –ÙôÒ¬¯ŒÌ gpu.0.gpu‚91.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚52.0 gpu.0.powerWatts‚118.933' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.644333333333336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UÞ¿2¢- _wandbruntime‚4635  _runtime‚46356.Ç£Á % _timestamp‚1.7844911586799963e+09 _runtime‚4635.254983368 _step‚723 & lr-AdamW/pg1‚0.0001910064772155321 & lr-AdamW/pg2‚0.0001910064772155321 trainer/global_step‚8949Y’ Ì¤Æ _runtime‚4635.254983368 _step‚723 & lr-AdamW/pg1‚0.0001910064772155321 & lr-AdamW/pg2‚0.0001910064772155321 trainer/global_step‚8949 % _timestamp‚1.7844911586799963e+09Ó®Ù´¥2¥- _wandbruntime‚4635  _runtime‚4635ìhñ¦âO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚694051527973376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚685453236030976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 160318783 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚694051886278144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚685453218512384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚678269084018800 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69799688 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8582851308598 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 _runtime‚ 4635.25597409 _step‚724 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90519095 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90508952 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚685453236030976 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69799686 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90519095 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71231566 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚544498802875904 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚686852292290080 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23704266 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160308638 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8862499019264 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90508952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23704559 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚544498827300352 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8598650247168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚553361301895168 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚678269101058160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8598309460992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚553361453198336 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47527302 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8583191231920 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7844911586810243e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚694051527973376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8598309460992 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚694051527973376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8598309460992 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71231566 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚544498802875904 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚686852292290080 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23704266 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160308638 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240Ô,-øÛ2¨- _wandbruntime‚4635  _runtime‚4635èãÙó&©íM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚685527410281472 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90518827 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚694127007965696 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚685527427800064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚304 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8599239372288 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71239437 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚685527427800064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47532381 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚694127007965696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47532383 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90518827 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚553421139775488 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69807190 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69807190 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8863585341440 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚686926289633114 _step‚725 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚544557703632896 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚678342527140640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8599239372288 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚553421288974336 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160326015 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23706752 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚302 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160336174 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71239133 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8584119455268 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160336174 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8949 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚694126649653760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160326015 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚685527410281472 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 % _timestamp‚1.7844911590572226e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90528984 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935¸ñ>rÔªòM YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 _runtime‚4635.632373149 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69807190 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8863585341440 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚686926289633114 _step‚725 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚544557703632896 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚678342527140640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8599239372288 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚553421288974336 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160326015 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23706752 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚302 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160336174 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71239133 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8584119455268 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160336174 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8949 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚694126649653760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160326015 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚685527410281472 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 % _timestamp‚1.7844911590572226e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90528984 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚685527410281472 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90518827 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚694127007965696 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚685527427800064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚304 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8599239372288 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71239437 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚685527427800064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47532381 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚694127007965696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47532383 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90518827 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚553421139775488 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69807190 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8910620142080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8629634296928 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚697807644249600 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8644837282816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91014697 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91004554 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 142914560 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚690569682516406 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚681940405182160 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚556353121155072 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 161179340 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 118490112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 trainer/global_step‚8999 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71610741 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚689162806966784 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚689162806966784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚547442619503104 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8644837282816 % _timestamp‚1.7844911826473415e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70174788 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71610446 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚556353264069632 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70174786 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚681940388142800 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70174786 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23829389 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚689162824485376 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚681940388142800 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70174786 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23829389 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚689162824485376 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8910620142080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8629634296928 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚697807644249600 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8644837282816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91014697 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91004554 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 142914560 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚690569682516406 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚681940405182160 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚556353121155072 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 161179340 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 118490112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 trainer/global_step‚8999 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71610741 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚689162806966784 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚689162806966784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚547442619503104 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8644837282816 % _timestamp‚1.7844911826473415e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70174788 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71610446 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚556353264069632 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70174786 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0Ø'¡u 2¶- _wandbruntime‚4659  _runtime‚4659¿Gà¾+·íM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚729 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 161196717 =7DeviceStatsMonitor.on_train_batcw›xÈh_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91024586 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8911584828416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8911457956864 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47786433 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91014429 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚682013831264640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 % _timestamp‚1.7844911833233426e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47786431 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚556412953792512 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8645767194112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚697883124241920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161206876 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71618006 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70182288 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 _runtime‚4659.898509634 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70182288 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8646107987456 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 161196717 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91024586 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8911584828416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8911457956864 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47786433 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91014429 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚682013831264640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 % _timestamp‚1.7844911833233426e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47786431 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚556412953792512 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8645767194112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚697883124241920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161206876 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71618006 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70182288 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 _runtime‚4659.898509634 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70182288 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8646107987456 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚729 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8691705890816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚701564118830592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8691705890816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚550386460554752 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚692872395421184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162060187 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚692872412939776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48035402 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23954714 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚732 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚685611709306160 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8675737438614 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8691365104640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162050042 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48035400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚559344868538880 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8958580258816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 147850240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚701563760525824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91510299 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8676077361936 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚701563760525824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8691365104640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0ÄØ¥¿ ÅçO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚685611709306160 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8675737438614 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8691365104640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162050042 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48035400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.large_^ÑP.pool.current‚675 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚559344868538880 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8958580258816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 147850240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚701563760525824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91510299 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8676077361936 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚701563760525824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8691365104640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70549888 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8691705890816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚701564118830592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8691705890816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚550386460554752 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚692872395421184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162060187 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚692872412939776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48035402 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23954714 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚732ÜØ27¥2Æ- _wandbruntime‚4684  _runtime‚4684B=Ibó&ÇíM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71997735 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚701638882206208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚692946604708864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70557388 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8692635809280 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 _runtime‚4684.416036405 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚559404860611072 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162067419 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70557390 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48040483 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚701639240518144 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚694361784011130 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70557388 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚692946587190272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23957254 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚685685118349280 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8692635809280 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8959392657920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162077578 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚685685135388640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162067419 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8692295015936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91510031 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚692946604708864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91520188 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70557390 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91510031 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 % _timestamp‚1.7844912078408425e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 162077578 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚550445336887296 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8692295015936 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71997735 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚701638882206208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚692946604708864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70557388 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8692635809280 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 _runtime‚4684.416036405 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚559404860611072 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162067419 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70557390 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48040483 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚701639240518144 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚694361784011130 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70557388 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚692946587190272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23957254 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚685685118349280 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8692635809280 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8959392657920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162077578 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚685685135388640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162067419 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8692295015936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91510031 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚692946604708864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91520188 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70557390 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91510031 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 % _timestamp‚1.7844912078408425e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 162077578 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚550445336887296 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8692295015936 FõWžÍ‚(:“ ÒÙôÒâïÄÌ gpu.0.gpu‚91.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚266.888' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.96266666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_¨ñ72Î- _wandbruntime‚4709  _runtime‚4709Ó=„‡ÇÏÁ & lr-AdamW/pg1‚0.0001902248839076814 & lr-AdamW/pg2‚0.0001902248839076814 trainer/global_step‚9099 % _timestamp‚1.7844912331186607e+09 _runtime‚4709.694609066 _step‚735°ÿÌÐÆ & lr-AdamW/pg2‚0.0001902248839076814 trainer/global_step‚9099 % _timestamp‚1.7844912331186607e+09 _runtime‚4709.694609066 _step‚735 & lr-AdamW/pg1‚0.0001902248839076814ßÇØÃ2Ñ- _wandbruntime‚4709  _runtime‚4709g?î? ÒâO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚705319876802048 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91995758 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24079967 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48289452 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8722520426944 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 Kác]Û BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚305 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚562336663145984 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8738233712640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91995758 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8738233712640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72369114 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚696581983875584 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚562336824934912 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚705319876802048 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚689282996390800 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9006547752960 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚698005176894422 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162920744 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 trainer/global_step‚9099 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70924988 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚696582001394176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8737892926464 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92005901 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚553330277181952 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚698005533857104 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 _step‚736 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 137364480 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48289450 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚553330252757504Z% í'ÓçO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91995758 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8738233712640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72369114 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚696581983875584 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚562336824934912 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚705319876802048 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚689282996390800 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9006547752960 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚698005176894422 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162920744 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 trainer/global_step‚9099 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70924988 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚696582001394176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8737892926464 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92005901 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚553330277181952 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚698005533857104 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 _step‚736 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 137364480 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48289450 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚553330252757504 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚705319876802048 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91995758 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24079967 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48289452 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8722520426944 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚305 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚562336663145984 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8738233712640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260àëŠWeÔ‚(:Ú áÙôÒ¢ò‚Ì& disk.nvme1n1.out‚13569.55029296875 disk.md0.out‚ 123.265625! disk.dm-1.out‚7079.5244140625 disk.rbd0.out‚ 0.4697265625* proc.memory.percent‚16.970443725585938 disk.nvme1n1.in‚0 disk.md0.in‚0# disk.dm-0.out‚48350.31689453125 disk.dm-1.in‚0 disk.rbd0.in‚0% disk.nvme0n1.out‚42125.3115234375 proc.cpu.threads‚17 network.sent‚ 1.2442075e+07 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375$ memory_percent‚96.57573699951172 cpu‚26.103546984988668 network.recv‚ 1.27582e+06) disk./.usagePercent‚76.59791351005282$ disk./.usageGB‚1175.969467163086 disk.nvme1c1n1.in‚0 disk.dm-0.in‚ 4.3359375' disk.nvme0c0n1.out‚42125.3115234375% proc.memory.availableMB‚ 561.03125 proc.memory.rssMB‚ 2780.4375( disk.nvme1c1n1.out‚13569.55029296875^7INžÕ‚(:“ áÙôÒ¸˜´Ì gpu.0.gpu‚55.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚247.441' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚82.48033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³þ 2Ö- _wandbruntime‚4710  _runtime‚4710îºÄÌò&×ìM E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9007500499456 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8738822837760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚696656193163264 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚696656175644672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92005633 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162948280 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 % _timestamp‚1.7844912334886734e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚705395356794368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚553389129090048 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9007375725056 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚698079531200138 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚689356422473280 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚696656175644672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚9099 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70932490 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70932488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚562396654013952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8723448650292 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 _runtime‚ 4710.06386876 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 162948280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72377040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚689356439512640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92015790 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92015790 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24082228 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8738822837760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8739163631104 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚705395356794368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚696656193163264 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162938121 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72376759 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0¤¶sJà ØñM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70932488 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92005633 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24082507 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚562396504815104 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚698079888162932 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8723108726858 _step‚737 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70932490 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚553389153514496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48294531 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72377040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚689356439512640 REDeviceStatsMonitor.on_train_batch_end/allocation.smalþ6ogl_pool.allocated‚92015790 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92015790 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24082228 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8738822837760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8739163631104 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚705395356794368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚696656193163264 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162938121 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72376759 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9007500499456 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8738822837760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚696656193163264 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚696656175644672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92005633 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162948280 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 % _timestamp‚1.7844912334886734e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚705395356794368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚553389129090048 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9007375725056 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚698079531200138 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚689356422473280 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚696656175644672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚9099 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70932490 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70932488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚562396654013952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8723448650292 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 _runtime‚ 4710.06386876 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 162948280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚709076351383040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚709075993078272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚709075993078272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8784420748288 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7844912590181866e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71300086 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚556274093809152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71300088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8784761534464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9054378979328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚700291572329984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 163801591 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48543502 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚701722924083430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚700291572329984 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8784761534464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8768623568630 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚565328591278592 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 118490112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚700291589848576 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚700291589848576 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24205646 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92501503 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92491360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 163791446 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71300086 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚701723281046112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48543500 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚4735.593185216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9054497469440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8768963491952 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8784420748288 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚709076351383040 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71300088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72749148 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 142914560 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72748871 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 trainer/global_step‚9149 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚692954317554160 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚565328448364032 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚692954317554160 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚565328448364032 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚709076351383040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚709075993078272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚709075993078272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8784420748288 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7844912590181866e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71300086 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚556274093809152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71300088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8784761534464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9054378979328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚700291572329984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 163801591 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48543502 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚701722924083430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚700291572329984 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8784761534464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8768623568630 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚565328591278592 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 118490112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚700291589848576 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚700291589848576 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24205646 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92501503 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92491360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 163791446 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71300086 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚701723281046112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48543500 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0ä}ƒj32ä- _wandbruntime‚4735  _runtime‚4735aAc¨2åíM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92511392 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚700365781617664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71307590 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚565388432047616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7844912593937986e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚709151114758656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9055341325824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 163808823 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚701797635351940 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163818982 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 145004544 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚741 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92501235 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 120580096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚693027743636640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚709151473070592 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚700365781617664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72756432 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8785350659584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8785691452928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚556332970141696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92511392 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71307588 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72756703 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚693027726597280 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163818982 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚701797278389146 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 _runtime‚4735.968948715 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8769891715300 trainer/global_step‚9149 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48548583 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24208120 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚556332945717248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚709151473070592 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92501235 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚565388287043072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8785350659584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48548581 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8785691452928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675iÚg.ø&æòM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92501235 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚565388287043072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8785350659584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48548581 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8785691452928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92511392 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚700365781617664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71307590 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚565388432047616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7844912593937986e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚709151114758656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9055341325824 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 163808823 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚701797635351940 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163818982 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 145004544 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚741 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92501235 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 120580096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚693027743636640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚709151473070592 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚700365781617664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72756432 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8785350659584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8785691452928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚556332970141696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92511392 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71307588 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72756703 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚693027726597280 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163818982 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚701797278389146 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 _runtime‚4735.968948715 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8769891715300 trainer/global_step‚9149 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48548583 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24208120 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚556332945717248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚709151473070592 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720å ôóy2ç- _wandbruntime‚4735  _runtime‚4735iZÄ è… %loss/total_loss‚8.208608594844378 _runtime‚4735.969542556 epoch‚11 2loss/SRqqt1/assignment_loss‚0.5213974823465054 2loss/SRqqt2/assignment_loss‚0.6627904936359528 /loss/FRt1/detection_loss‚1.2299170276846003 /loss/FRt2/detection_loss‚0.5049414531563543 _step‚742 1loss/SRqqt1/detection_loss‚0.6074339202994377 trainer/global_step‚9149 0loss/FRt1/assignment_loss‚0.7966993772407447 0loss/FRt2/assignment_loss‚1.7381660864538118 4loss/FBt2/assignment_loss‚0.00035875431660308265 /loss/FBt1/detection_loss‚0.6252757525313404 $ _timestamp‚1.784491259394891e+09 2loss/FBt1/assignment_loss‚0.007319424593291784 1loss/SRqqt2/detection_loss‚0.9153233329416253 /loss/FBt2/detection_loss‚0.5989854896441102ëjlAéŠ 0loss/FRt1/assignment_loss‚0.7966993772407447 0loss/FRt2/assignment_loss‚1.7381660864538118 4loss/FBt2/assignment_loss‚0.00035875431660308265 /loss/FBt1/detection_loss‚0.6252757525313404 $ _timestamp‚1.784491259394891e+09 2loss/FBt1/assignment_loss‚0.007319424593291784 1loss/SRqqt2/detection_loss‚0.9153233329416253 /loss/FBt2/detection_loss‚0.5989854896441102 %loss/total_loss‚8.208608594844378 _runtime‚4735.969542556 epoch‚11 2loss/SRqqt1/assignment_loss‚0.5213974823465054 2loss/SRqqt2/assignment_loss‚0.6627904936359528 /loss/FRt1/detection_loss‚1.2299170276846003 /loss/FRt2/detection_loss‚0.5049414531563543 _step‚742 1loss/SRqqt1/detection_loss‚0.6074339202994377 trainer/global_step‚9149æ0;îkê‚(:à ÿÙôÒ·ãÒË disk.nvme0n1.in‚ 4.3359375 disk.md0.out‚ 123.8984375 disk.dm-0.in‚ 4.3359375 disk.dm-0.out‚48360.30859375& disk.nvme0n1.out‚42131.90576171875( disk.nvme1c1n1.out‚13574.71728515625 disk.nvme1n1.in‚0& disk.nvme1n1.out‚13574.71728515625$ memory_percent‚96.61822319030762) proc.memory.percent‚17.00921058654785 proc.cpu.threads‚17% disk./.usageGB‚1175.9693565368652 disk.rbd0.in‚0 cpu‚26.105422264361135 network.recv‚ 1.284192e+06 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.dm-1.in‚0! disk.dm-1.out‚7079.9326171875' proc.memory.availableMB‚ 554.0703125" proc.memory.rssMB‚ 2786.7890625 network.sent‚ 1.2532484e+07) disk./.usagePercent‚76.59790630430649 disk.rbd0.out‚ 0.4697265625( disk.nvme0c0n1.out‚42131.90576171875Ai¬.œë‚(:‘ ÿÙôÒÉŸÌË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚265.539' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.51299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03ôVkì‚(:à ŽÚôÒõÆàË( disk.nvme0c0n1.out‚42132.80419921875& disk.nvme0n1.out‚42132.80419921875 disk.nvme1c1n1.in‚0( proc.memory.availableMB‚ 554.55859375* proc.memory.percent‚17.009258270263672 proc.cpu.threads‚17 network.sent‚ 1.2596621e+07 disk.nvme1n1.in‚0& disk.nvme1n1.out‚13575.29443359375 disk.nvme0c0n1.in‚ 4.3359375) disk./.usagePercent‚76.59790257719631% disk./.usageGB‚1175.9692993164062 disk.md0.out‚ 124.2109375 disk.nvme0n1.in‚ 4.3359375! proc.memory.rssMB‚ 2786.796875 cpu‚26.106504702172025 network.recv‚ 1.288745e+06 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 disk.dm-0.out‚48360.94921875 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625( disk.nvme1c1n1.out‚13575.29443359375$ memory_percent‚96.61524295806885 disk.dm-1.in‚0 disk.dm-1.out‚7080.087890625òöìží‚(:“ ŽÚôÒ‡üœÌ gpu.0.gpu‚76.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚263.684' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.89466666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$ê˜î2î- _wandbruntime‚4761  _runtime‚4761h Ð¥Éïà _step‚743 ' lr-AdamW/pg1‚0.00018969710359160874 ' lr-AdamW/pg2‚0.00018969710359160874 trainer/global_step‚9199 % _timestamp‚1.7844912850498977e+09 _runtime‚4761.624868057ÈàùÎðÈ trainer/global_step‚9199 % _timestamp‚1.7844912850498977e+09 _runtime‚4761.624868057 _step‚743 ' lr-AdamW/pg1‚0.00018969710359160874 ' lr-AdamW/pg2‚0.00018969710359160874缚ªÍ2ñ- _wandbruntime‚4761  _runtime‚4761,ysÑ%òãO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92997105 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8830948570112 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73128085 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚705440671272438 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚712832467659264 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8815406556960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71675186 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48797550 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚704001178302976 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73128369 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚568320421005824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 164662148 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92997105 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚568320271799808 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚559217910436352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8831289356288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48797552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24330535 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8831289356288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚712832467659264 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124781568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164672293 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚705441028235120 trainer/global_step‚9199 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9102385787904 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚704001160784384 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71675186 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8815066633638 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚559217886011904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7844912850509148e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9102510569472 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚696625604638800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚704001178302976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92986962 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24330817 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚704001160784384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚712832109354496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚712832109354496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8830948570112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peì£Â\ak‚ 13862174720 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71675188 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240/8c î'óèO _runtime‚4761.625944129 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92986962 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9102510569472 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚696625604638800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚704001178302976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92986962 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24330817 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚704001160784384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚712832109354496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚712832109354496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8830948570112 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71675188 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92997105 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8830948570112 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73128085 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚705440671272438 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚712832467659264 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8815406556960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71675186 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48797550 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚704001178302976 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73128369 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚568320421005824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 164662148 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92997105 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚568320271799808 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚559217910436352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8831289356288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48797552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24330535 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8831289356288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚712832467659264 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124781568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164672293 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚705441028235120 trainer/global_step‚9199 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9102385787904 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚704001160784384 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71675186 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8815066633638 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚559217886011904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7844912850509148e+09 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8832219274752 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71682690 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164679525 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71682690 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8831878481408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24333309 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚745 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8832219274752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93006994 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8815994856874 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71682688 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚559276786768896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73135658 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9103474767360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚568380261536256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚696699030721280 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164679525 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚712907589346816 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚704075352553472 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48802633 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164689684 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675Ñø&öòM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚559276762344448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9103345798656 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8816334780308 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 _runtime‚4762.334478182 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8832219274752 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71682690 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164679525 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71682690 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8831878481408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24333309 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚745 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8832219274752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93006994 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8815994856874 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71682688 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚559276786768896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73135658 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9103474767360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚568380261536256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚696699030721280 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164679525 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚712907589346816 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚704075352553472 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48802633 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164689684 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚712907231034880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374é$h8n2÷- _wandbruntime‚4762  _runtime‚4762ÇO¥k ø… 0loss/FRt1/assignment_loss‚0.8139577624388039 1loss/SRqqt2/assignment_loss‚0.509030194781671 /loss/FRt1/detection_loss‚1.3224940890081598 /loss/FBt2/detection_loss‚0.7525492536232752 trainer/global_step‚9199 % _timestamp‚1.7844912857604578e+09 3loss/FBt2/assignment_loss‚0.0001409996687154175 epoch‚11 3loss/FBt1/assignment_loss‚0.0010148481676276635 /loss/FRt2/detection_loss‚0.5265626175670028 1loss/SRqqt2/detection_loss‚1.1747903363532661 /loss/FBt1/detection_loss‚0.4005641957328683 _runtime‚4762.335196594 _step‚746 0loss/FRt2/assignment_loss‚1.8740907398389208 2loss/SRqqt1/assignment_loss‚0.5201500676851655 1loss/SRqqt1/detection_loss‚0.5647644746109489 %loss/total_loss‚8.460109579476427Ï9z}ùŠ 0loss/FRt1/assignment_loss‚0.8139577624388039 1loss/SRqqt2/assignment_loss‚0.509030194781671 /loss/FRt1/detection_loss‚1.3224940890081598 /loss/FBt2/detection_loss‚0.7525492536232752 trainer/global_step‚9199 % _timestamp‚1.7844912857604578e+09 3loss/FBt2/assignment_loss‚0.0001409996687154175 epoch‚11 3loss/FBt1/assignment_loss‚0.0010148481676276635 /loss/FRt2/detection_loss‚0.5265626175670028 1loss/SRqqt2/detection_loss‚1.1747903363532661 /loss/FBt1/detection_loss‚0.4005641957328683 _runtime‚4762.335196594 _step‚746 0loss/FRt2/assignment_loss‚1.8740907398389208 2loss/SRqqt1/assignment_loss‚0.5201500676851655 1loss/SRqqt1/detection_loss‚0.5647644746109489 %loss/total_loss‚8.460109579476427êOgŽ ú‚(:ã ÚôÒ´„¿Ì disk.dm-0.in‚ 4.3359375$ memory_percent‚96.61915302276611( proc.memory.availableMB‚ 553.91796875 disk.nvme0n1.in‚ 4.3359375 network.sent‚ 1.2599644e+07 network.recv‚ 1.292173e+06% disk./.usageGB‚1175.9692687988281 D»L¶a disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375' disk.nvme0c0n1.out‚42134.1806640625 disk.md0.in‚0# disk.dm-0.out‚48361.96826171875 disk.dm-1.out‚7080.189453125# proc.memory.rssMB‚ 2787.40234375 cpu‚26.108461670969614 proc.cpu.threads‚17% disk.nvme0n1.out‚42134.1806640625 disk.nvme1n1.in‚0& disk.nvme1n1.out‚13575.88623046875( disk./.usagePercent‚76.5979005894042 disk.rbd0.out‚ 0.4697265625 disk.md0.out‚ 124.60546875 disk.dm-1.in‚0* proc.memory.percent‚17.012953758239746 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚13575.88623046875¹à‰žû‚(:“ ÚôÒ¶ýôÌ gpu.0.gpu‚52.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚251.048' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚83.68266666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MB¤±iü‚(:Þ ¬ÚôÒ²çäË network.recv‚ 1.296616e+06) disk./.usagePercent‚76.59792096427317% disk.nvme0n1.out‚42135.4462890625 disk.nvme1n1.in‚0 disk.md0.in‚0" disk.dm-1.out‚7080.49658203125# proc.memory.rssMB‚ 2787.42578125& disk.nvme1c1n1.out‚13576.775390625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375' disk.nvme0c0n1.out‚42135.4462890625 disk.nvme0n1.in‚ 4.3359375$ disk.nvme1n1.out‚13576.775390625 disk.md0.out‚ 125.015625# disk.dm-0.out‚48362.93310546875( proc.memory.availableMB‚ 553.61328125$ disk./.usageGB‚1175.969581604004 disk.rbd0.out‚ 0.4697265625 disk.nvme1c1n1.in‚0 disk.dm-0.in‚ 4.3359375# memory_percent‚96.6210126876831* proc.memory.percent‚17.013096809387207 cpu‚26.110579166895594 proc.cpu.threads‚17 network.sent‚ 1.2663747e+07v¨¶“ý‚(:ˆ ¬ÚôÒ´ö–Ì gpu.0.gpu‚72.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚265.995' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚88.665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡/+V2þ- _wandbruntime‚4787  _runtime‚4787ñ§àMÉÿà % _timestamp‚1.7844913112140431e+09 _runtime‚4787.789061945 _step‚747 ' lr-AdamW/pg1‚0.00018943121064830286 ' lr-AdamW/pg2‚0.00018943121064830286 trainer/global_step‚9249<$÷æÎ€È trainer/global_step‚9249 % _timestamp‚1.7844913112140431e+09 _runtime‚4787.789061945 _step‚747 ' lr-AdamW/pg1‚0.00018943121064830286 ' lr-AdamW/pg2‚0.00018943121064830286ëOvÊ­2- _wandbruntime‚4787  _runtime‚4787‹ÆÛwé'‚ãO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7844913112151973e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49051602 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165532850 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93482564 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8861849621968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8877817178112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165542995 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9150473066496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚571312200130048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9150344090624 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8877476391936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚707710766757376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24455373 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93482564 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚707710749238784 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72050286 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 trainer/global_step‚9249 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72050288 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8861509698646 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8877817178112 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 _step‚748 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93492707 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚562161702639104 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49051600 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚707710749238784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚571312046729728 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚700296925802160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72050286 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚716588225630720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8877476391936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73507254 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _runtime‚4787.790221797 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚562161727063552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚709158418461446 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚716588225630720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚716588583935488 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935–KÎBî'ƒèO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7844913112151973e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49051602 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165532850 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93482564 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8861849621968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8877817178112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165542995 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9150473066496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚571312200130048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9150344090624 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8877476391936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚707710766757376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24455373 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93482564 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚707710749238784 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72050286 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 trainer/global_step‚9249 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72050288 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8861509698646 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8877817178112 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 _step‚748 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93492707 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚562161702639104 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49051600 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚707710749238784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚571312046729728 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚700296925802160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72050286 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚716588225630720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8877476391936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73507254 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _runtime‚4787.790221797 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚562161727063552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚709158418461446 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚716588225630720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚716588583935488ìc‚Až2„- _wandbruntime‚4788  _runtime‚4788|Œí­$…ìM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚716663705623040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165550227 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚700370334845280 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚716663705623040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24458146 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚571372040002048 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚562220603396096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚707784958526464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24457864 trainer/global_step‚9249 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72057790 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8862437921882 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93502596 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8862777845316 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 _runtime‚4788.164574022 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8878747096576 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72057788 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73514829 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49056681 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚709232772767162 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72057790 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚707784941007872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚707784941007872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73514545 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9151436605952 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165550227 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93492439 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚707784958526464 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8878406303232 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165560386 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚749 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49056683 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165560386 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93502596 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚571371884511744 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚lцE562220578971648 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚716663347311104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93492439 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚707784941007872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73514545 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9151436605952 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165550227 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93492439 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚707784958526464 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8878406303232 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165560386 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚749 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49056683 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165560386 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93502596 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚571371884511744 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚562220578971648 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚716663347311104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93492439 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚716663705623040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165550227 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚700370334845280 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚716663705623040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24458146 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚571372040002048 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚562220603396096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚707784958526464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24457864 trainer/global_step‚9249 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72057790 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8862437921882 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93502596 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8862777845316 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 _runtime‚4788.164574022 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8878747096576 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72057788 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73514829 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49056681 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚709232772767162 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448íõØÔ2‡- _wandbruntime‚4788  _runtime‚4788žê|óˆ _step‚750 /loss/FRt2/detection_loss‚0.5237793519334547 1loss/SRqqt1/detection_loss‚0.5839315794873983 %loss/total_loss‚8.734730306318133 trainer/global_step‚9249 % _timestamp‚1.7844913115904834e+09 2loss/SRqqt2/assignment_loss‚0.5573004267544291 1loss/FBt2/assignment_loss‚0.04585524644067608 0loss/SRqqt2/detection_loss‚1.038376903025447 _runtime‚4788.165725844 0loss/FRt1/assignment_loss‚0.9069404370718069 1loss/SRqqt1/assignment_loss‚0.508005240221187 2loss/FBt1/assignment_loss‚0.020418530122698543 epoch‚11 0loss/FRt2/assignment_loss‚1.9313977125376591 /loss/FRt1/detection_loss‚1.2286688790549165 /loss/FBt1/detection_loss‚0.6911769343181872 /loss/FBt2/detection_loss‚0.6988790653502712_…ô$ ‰† /loss/FRt2/detection_loss‚0.5237793519334547 1loss/SRqqt1/detection_loss‚0.5839315794873983 %loss/total_loss‚8.734730306318133 trainer/global_step‚9249 % _timestamp‚1.7844913115904834e+09 2loss/SRqqt2/assignment_loss‚0.5573004267544291 1loss/FBt2/assignment_loss‚0.04585524644067608 0loss/SRqqt2/detection_loss‚1.038376903025447 _runtime‚4788.165725844 0loss/FRt1/assignment_loss‚0.9069404370718069 1loss/SRqqt1/assignment_loss‚0.508005240221187 2loss/FBt1/assignment_loss‚0.020418530122698543 epoch‚11 0loss/FRt2/assignment_loss‚1.9313977125376591 /loss/FRt1/detection_loss‚1.2286688790549165 /loss/FBt1/detection_loss‚0.6911769343181872 /loss/FBt2/detection_loss‚0.6988790653502712 _step‚750îU8tœŠ‚(:‘ »ÚôÒÇô³Ë gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚215.542' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚71.84733333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯ä@Ìi‹‚(:Þ »ÚôÒ€½¬Ì* proc.memory.percent‚17.015695571899414 cpu‚26.112720991651926 proc.cpu.threads‚17 network.sent‚ 1.2666711e+07 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚48393.32177734375" disk.dm-1.out‚7080.62158203125 disk.rbd0.out‚ 0.4697265625' disk.nvme0c0n1.out‚42165.8623046875 disk.nvme0n1.in‚ 4.3359375% disk.nvme0n1.out‚42165.8623046875 network.recv‚ 1.299687e+06) disk./.usagePercent‚76.51943696902097 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚13577.380859375 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚13577.380859375# memory_percent‚96.6231107711792% disk./.usageGB‚1174.7646560668945 disk.md0.out‚ 125.234375 disk.nvme0c0n1.in‚ 4.3359375( proc.memory.availableMB‚ 553.26953125" proc.memory.rssMB‚ 2787.8515625 disk.md0.in‚0 disk.rbd0.in‚0QÃ*&2Œ- _wandbruntime‚4813  _runtime‚4813Ó^tdÉà % _timestamp‚1.7844913372889242e+09 _runtime‚4813.863925331 _step‚751 ' lr-AdamW/pg1‚0.00018916398818611635 ' lr-AdamW/pg2‚0.00018916398818611635 trainer/global_step‚9299eþb%ÎŽÈ _runtime‚4813.863925331 _step‚751 ' lr-AdamW/pg1‚0.00018916398818611635 ' lr-AdamW/pg2‚0.00018916398818611635 trainer/global_step‚9299 % _timestamp‚1.7844913372889242e+09ï%´³l2- _wandbruntime‚4813  _runtime‚4813I!µé'ãO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚565105543690752 _runtime‚4813.865014422 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚711420355211776 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93988309 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8924344999936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚574303994724352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚565105519266304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚711420337693184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚711420337693184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49305650 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8924004213760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚703968212886800 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 % _timestamp‚1.7844913372900264e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚720344341906944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24580427 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72425386 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚703968229926160 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚720344700211712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8924004213760 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚712876165650454 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72425388 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8907952763654 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 166403552 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8908292686976 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9198451033600 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72425386 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚720344341906944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93978166 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73886079 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9198317863424 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8924344999936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚574303837129728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93988309 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚711420355211776 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 166413697 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24580137 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚712876522613136 trainer/global_step‚9299 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚720344700211712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8924344999936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚574303994724352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚565105519266304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚711420337693184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚711420337693184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49305650 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8924004213760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚703968212886800 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 % _timestamp‚1.7844913372900264e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚720344341906944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24580427 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72425386 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚703968229926160 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚720344700211712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8924004213760 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚712876165650454 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72425388 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8907952763654 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 166403552 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8908292686976 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9198451033600 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72425386 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚720344341906944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93978166 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73886079 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9198317863424 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8924344999936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚574303837129728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93988309 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚711420355211776 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 166413697 TGDeviceStatsMonitor.on_train_batch_st{Öܳart/inactive_split.small_pool.freed‚24580137 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚712876522613136 trainer/global_step‚9299 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚720344700211712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 166431088 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93988041 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚711494529462272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49310733 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72432888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24582896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93998198 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚720419821899264 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73893353 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 trainer/global_step‚9299 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72432890 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚704041656008640 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚574363828358144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73893629 % _timestamp‚1.7844913376654522e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 166420929 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93988041 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚720419463587328 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 166420929 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8909220910324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8925274918400 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚704041638969280 _step‚753 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8924934125056 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49310731 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚712950519956170 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 166431088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8924934125056 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚565164395598848 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚711494546980864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9199273074688 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24582622 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2w7š÷&”ñM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 166420929 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93988041 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚720419463587328 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 166420929 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8909220910324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8925274918400 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚704041638969280 _step‚753 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8924934125056 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49310731 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚712950519956170 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 166431088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8924934125056 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚565164395598848 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚711494546980864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9199273074688 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24582622 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8925274918400 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 166431088 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93988041 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚711494529462272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49310733 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72432888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24582896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93998198 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚720419821899264 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73893353 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 trainer/global_step‚9299 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72432890 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚704041656008640 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚574363828358144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73893629 % _timestamp‚1.7844913376654522e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2ñn ï2•- _wandbruntime‚4814  _runtime‚4814– –ƒ 0loss/FRt2/assignment_loss‚1.8462630834495812 2loss/SRqqt1/assignment_loss‚0.5559627015065024 3loss/FBt1/assignment_loss‚0.0005287398913830544 1loss/FBt2/assignment_loss‚0.01920580044320833 /loss/FRt2/detection_loss‚0.5252804948698038 /loss/FBt2/detection_loss‚0.6444717368410368 1loss/SRqqt2/assignment_loss‚0.568317196535645 /loss/FRt1/detection_loss‚1.3597734719514847 1loss/SRqqt1/detection_loss‚0.5835003048815459 1loss/SRqqt2/detection_loss‚1.0502892557867907 % _timestamp‚1.7844913376665273e+09 0loss/FRt1/assignment_loss‚0.8762150334787893 %loss/total_loss‚8.786044041817242 epoch‚11 _runtime‚4814.244413936 _step‚754 /loss/FBt1/detection_loss‚0.7562362221814692 trainer/global_step‚9299ˆ˜—ˆ 0loss/FRt2/assignment_loss‚1.8462630834495812 2loss/SRqqt1/assignment_loss‚0.5559627015065024 3loss/FBt1/assignment_loss‚0.0005287398913830544 1loss/FBt2/assignment_loss‚0.01920580044320833 /loss/FRt2/detection_loss‚0.5252804948698038 /loss/FBt2/detection_loss‚0.6444717368410368 1loss/SRqqt2/assignment_loss‚0.568317196535645 /loss/FRt1/detection_loss‚1.3597734719514847 1loss/SRqqt1/detection_loss‚0.5835003048815459 1loss/SRqqt2/detection_loss‚1.0502892557867907 % _timestamp‚1.7844913376665273e+09 0loss/FRt1/assignment_loss‚0.8762150334787893 %loss/total_loss‚8.786044041817242 epoch‚11 _runtime‚4814.244413936 _step‚754 /loss/FBt1/detection_loss‚0.7562362221814692 trainer/global_step‚9299òƒösmf˜‚(:Û ÊÚôÒœØïË$ disk./.usageGB‚1174.764560699463 disk.nvme1n1.in‚0 disk.md0.in‚0! disk.dm-1.out‚7080.7763671875* proc.memory.percent‚17.017650604248047 network.sent‚ 1.272936e+07 network.recv‚ 1.303992e+06( disk.nvme1c1n1.out‚13577.99365234375& disk.nvme1n1.out‚13577.99365234375 disk.dm-0.in‚ 4.3359375% disk.nvme0c0n1.out‚42166.79296875$ memory_percent‚96.62604331970215' proc.memory.availableMB‚ 552.7890625) disk./.usagePercent‚76.51943075717068 disk.md0.out‚ 125.5859375 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375# disk.nvme0n1.out‚42166.79296875! proc.memory.rssMB‚ 2788.171875 disk.nvme1c1n1.in‚0# disk.dm-0.out‚48393.94287109375 disk.dm-1.in‚0 cpu‚26.114913086967377 proc.cpu.threads‚17´Z|Áž™‚(:“ ÊÚôÒ³ªæË gpu.0.gpu‚74.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚246.992' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚82.33066666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O±¹nš‚(:ã ÙÚôÒ´ýÞË network.sent‚ 1.279131e+07) disk./.usagePercent‚76.51946380421425& disk.nvme1n1.out‚13578.72216796875 disk.md0.out‚ 125.93359375 disk.dm-1.in‚0! disk.dm-1.out‚7080.8583984375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375" disk.dm-0.out‚48397.0751953125 disk.rbd0.out‚ 0.4697265625( disk.nvme0c0n1.out‚42170.02490234375 disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚42170.02490234375( proc.memory.availableMB‚ 552.28515625 proc.cpu.threads‚17 network.recv‚ 1.307757e+06% disk./.usageGB‚1174.7650680541992 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 disk.nvme1c1n1.in‚0$ memory_percent‚96.62911891937256" proc.memory.rssMB‚ 2788.4765625) proc.memory.percent‚17.01951026916504 disk.nvme1n1.in‚0( disk.nvme1c1n1.out‚13578.72216796875 cpu‚26.117087643249363¼J§%ž›‚(:“ ÙÚôÒì„áË gpu.0.gpu‚95.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚243.499' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚81.16633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó5§ß2œ- _wandbruntime‚4839  _runtime‚4839o(Éà trainer/global_step‚9349 % _timestamp‚1.7844913633459864e+09 _runtime‚4839.921014028 _step‚755 ' lr-AdamW/pg1‚0.00018889544048515946 ' lr-AdamW/pg2‚0.00018889544048515946­M« ÎžÈ _runtime‚4839.921014028 _step‚755 ' lr-AdamW/pg1‚0.00018889544048515946 ' lr-AdamW/pg2‚0.00018889544048515946 trainer/global_step‚9349 % _timestamp‚1.7844913633459864e+09ó‡B>•2Ÿ- _wandbruntime‚4839  _runtime‚4839Œ´qn âO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94483911 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94473768 _step‚756 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 120587264 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49559700 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚724100458183168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8954395828662 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚724100458183168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚715129943666176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚715129943666176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94473768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚715129926147584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9246275678208 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72800486 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 _runtime‚ 4839.92203492 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74264501 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚568049335893504 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 WGýÙzDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74264780 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8970872821760 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72800488 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚724100816487936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8970872821760 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74264780 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8970872821760 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72800488 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚724100816487936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8970872821760 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49559700 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚724100458183168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8954395828662 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚724100458183168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚715129943666176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚715129943666176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94473768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚715129926147584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9246275678208 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72800486 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 _runtime‚ 4839.92203492 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784ô ‘>2¢- _wandbruntime‚4840  _runtime‚4840-ÈY2ò&£ìM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72807988 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94483643 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167291631 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74272323 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 167291631 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72807988 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167301790 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8971461946880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49564783 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚716668624107972 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚724175938175488 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚568108212226048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24707540 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94493800 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚715204135435264 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚724175579863552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8955663975332 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 _step‚757 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8971802740224 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚568108236650496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚707712960132640 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8955324051898 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚715204117916672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72807990 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94493800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24707264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94483643 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚577355596271104 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 trainer/global_step‚9349 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74272045 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8971802740224 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚715204117916672 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167301790•s”ªR!¤ñM EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74272045 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8971802740224 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚715204117916672 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167301790 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72807988 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94483643 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167291631 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74272323 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 167291631 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72807988 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167301790 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8971461946880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49564783 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚716668624107972 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚724175938175488 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚568108212226048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24707540 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94493800 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚715204135435264 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚724175579863552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8955663975332 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 _step‚757 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8971802740224 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚568108236650496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_byteí¢x¤¥s.small_pool.current‚ 126871552 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚707712960132640 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8955324051898 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚715204117916672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72807990 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94493800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24707264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94483643 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚577355596271104 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 trainer/global_step‚9349 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720õ÷Po$2¥- _wandbruntime‚4840  _runtime‚4840oƒ ¦† epoch‚11 0loss/FRt2/assignment_loss‚1.9257151576889655 1loss/SRqqt2/detection_loss‚0.8684891780717995 /loss/FBt1/detection_loss‚0.3770206568419541 /loss/FRt1/detection_loss‚1.3697260817407502 /loss/FRt2/detection_loss‚0.5301712917362278 trainer/global_step‚9349 _runtime‚4840.297449979 /loss/FBt2/detection_loss‚0.4693490688999494 0loss/FRt1/assignment_loss‚0.9315970039715732 2loss/SRqqt2/assignment_loss‚0.6183146682455439 3loss/FBt1/assignment_loss‚0.0017600706428913333 4loss/FBt2/assignment_loss‚0.00032776519719964387 0loss/SRqqt1/detection_loss‚0.560292202011407 _step‚758 2loss/SRqqt1/assignment_loss‚0.5749205697379829 %loss/total_loss‚8.227683714786245 % _timestamp‚1.7844913637223508e+09n¹ò§‹ epoch‚11 0loss/FRt2/assignment_loss‚1.9257151576889655 1loss/SRqqt2/detection_loss‚0.8684891780717995 /loss/FBt1/detection_loss‚0.3770206568419541 /loss/FRt1/detection_loss‚1.3697260817407502 /loss/FRt2/detection_loss‚0.5301712917362278 trainer/global_step‚9349 _runtime‚4840.297449979 /loss/FBt2/detection_loss‚0.4693490688999494 0loss/FRt1/assignment_loss‚0.9315970039715732 2loss/SRqqt2/assignment_loss‚0.6183146682455439 3loss/FBt1/assignment_loss‚0.0017600706428913333 4loss/FBt2/assignment_loss‚0.00032776519719964387 0loss/SRqqt1/detection_loss‚0.560292202011407 _step‚758 2loss/SRqqt1/assignment_loss‚0.5749205697379829 %loss/total_loss‚8.227683714786245 % _timestamp‚1.7844913637223508e+09öáËyž¨‚(:“ èÚôÒ‰“‘Ì gpu.0.gpu‚91.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚183.859' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚61.28633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹dÌ m©‚(:â èÚôÒÈʶÌ# proc.memory.rssMB‚ 2789.09765625 cpu‚26.119136695950342" disk.dm-1.out‚7081.08935546875 network.sent‚ 1.2808977e+07 network.recv‚ 1.311136e+06% disk./.usageGB‚1170.7103271484375' disk.nvme1c1n1.out‚13580.1005859375 disk.md0.in‚0 disk.dm-1.in‚0( disk.nvme0c0n1.out‚42171.91845703125 disk.dm-0.in‚ 4.3359375" disk.dm-0.out‚48398.3134765625 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375* proc.memory.percent‚17.023301124572754) disk./.usagePercent‚76.25535431678473 disk.md0.out‚ 126.8046875& disk.nvme0n1.out‚42171.91845703125& proc.memory.availableMB‚ 551.796875 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0% disk.nvme1n1.out‚13580.1005859375 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625$ memory_percent‚96.63209915161133Ûœ¡$hª‚(:Ý ÷ÚôÒî´åË% disk.nvme0c0n1.out‚42178.62890625 disk.nvme0n1.in‚ 4.3359375 disk.nvme1n1.in‚0 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚48405.11572265625( proc.memory.availableMB‚ 552.05078125 proc.cpu.threads‚17 disk.rbd0.out‚ 0.4697265625$ memory_percent‚96.63054943084717 network.recv‚ 1.314879e+06) disk./.usagePercent‚76.25538090350399 disk.nvme0c0n1.in‚ 4.3359375# disk.nvme0n1.out‚42178.62890625 disk.nvme1c1n1.in‚0" disk.dm-1.out‚7081.14404296875 disk.rbd0.in‚0* proc.memory.percent‚17.023301124572754 network.sent‚ 1.2860212e+07$ disk./.usageGB‚1170.710735321045' disk.nvme1c1n1.out‚13581.0166015625 disk.md0.in‚0 disk.md0.out‚ 127.171875 disk.dm-1.in‚0# proc.memory.rssMB‚ 2789.09765625 cpu‚26.120931494436807% disk.nvme1n1.out‚13581.0166015625[|ùØ’«‚(:‡ ÷ÚôÒ´‘¡Ì gpu.0.gpu‚43.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚260.796' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚86.932 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•U‡2¬- _wandbruntime‚4865  _runtime‚4865_lÉ­Ã ' lr-AdamW/pg1‚0.00018862557184676863 ' lr-AdamW/pg2‚0.00018862557184676863 trainer/global_step‚9399 % _timestamp‚1.7844913893397567e+09 _runtime‚4865.914770715 _step‚759eáyÎ®È ' lr-AdamW/pg2‚0.00018862557184676863 trainer/global_step‚9399 % _timestamp‚1.7844913893397567e+09 _runtime‚4865.914770715 _step‚759 ' lr-AdamW/pg1‚0.00018862557184676863÷á"[Í2¯- _wandbruntime‚4865  _runtime‚4865‚M…wé'°ãO VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7844913893408456e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚727856574459392 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73175588 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94979513 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚580287387868160 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚711310838174160 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9000838893670 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73175588 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚727856574459392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49813750 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24831212 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74644964 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9017400643584 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚718839514601984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49813752ÅÌæ›î'±èO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73175588 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94979513 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚580287387868160 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚711310838174160 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9000838893670 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73175588 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚727856574459392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49813750 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24831212 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74644964 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9017400643584 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚718839514601984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49813752 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7844913893408456e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚727856574459392 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145øn2²- _wandbruntime‚4866  _runtime‚4866Ù“³“³íM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚718913723889664 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚711384264256640 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚571052028853248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚711384247217280 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73183088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24833432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9001767116906 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94979245 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚302 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9295198990336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 168162333 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73183088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚300 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚720386371296980 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚720386014334186 _step‚761 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚727931696139776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844913897142363e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168172492 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚727932054451712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73183090 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚718913706371072 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚580347227843584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonÂØMw`itor.on_train_batch_end/segment.large_pool.current‚675 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9018330562048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9017989768704 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73183090 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94979245 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9002107040340 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚727932054451712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74652565 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚718913723889664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9018330562048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74652263 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 168162333 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚571052053277696 _runtime‚4866.289432816 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 168172492 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24833732W‘‰ø&´òM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9018330562048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9017989768704 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73183090 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94979245 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9002107040340 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚727932054451712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74652565 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚718913723889664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9018330562048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74652263 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 168162333 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚571052053277696 _runtime‚4866.289432816 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 168172492 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24833732 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚718913723889664 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚711384264256640 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚571052028853248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚711384247217280 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73183088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24833432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9001767116906 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94979245 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚302 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9295198990336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 168162333 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73183088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚300 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚720386371296980 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚720386014334186 _step‚761 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚727931696139776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844913897142363e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168172492 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚727932054451712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73183090 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚718913706371072 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚580347227843584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360ù•×b2µ- _wandbruntime‚4866  _runtime‚4866èê>¶€ 0loss/FRt1/assignment_loss‚0.9055462606409763 /loss/FRt1/detection_loss‚1.3150256710384596 trainer/global_step‚9399 % _timestamp‚1.7844913897153828e+09 2loss/SRqqt1/assignment_loss‚0.5123320633543119 1loss/FBt2/assignment_loss‚0.00861140472776587 "loss/total_loss‚8.512870224838 _runtime‚4866.290093187 _step‚762 /loss/FRt2/detection_loss‚0.5708505644777812 1loss/SRqqt2/detection_loss‚0.8526683567486555 /loss/FBt1/detection_loss‚0.7629029835672834 /loss/FBt2/detection_loss‚0.6006411696174021 epoch‚11 0loss/FRt2/assignment_loss‚1.8390824175992255 2loss/SRqqt2/assignment_loss‚0.5239498863769202 2loss/FBt1/assignment_loss‚0.001705316028267207 1loss/SRqqt1/detection_loss‚0.6195541306609528ð…â ·… % _timestamp‚1.7844913897153828e+09 2loss/SRqqt1/assignment_loss‚0.5123320633543119 1loss/FBt2/assignment_loss‚0.00861140472776587 "loss/total_loss‚8.512870224838 _runtime‚4866.290093187 _step‚762 /loss/FRt2/detection_loss‚0.5708505644777812 1loss/SRqqt2/detection_loss‚0.8526683567486555 /loss/FBt1/detection_loss‚0.7629029835672834 /loss/FBt2/detection_loss‚0.6006411696174021 epoch‚11 0loss/FRt2/assignment_loss‚1.8390824175992255 2loss/SRqqt2/assignment_loss‚0.5239498863769202 2loss/FBt1/assignment_loss‚0.001705316028267207 1loss/SRqqt1/detection_loss‚0.6195541306609528 0loss/FRt1/assignment_loss‚0.9055462606409763 /loss/FRt1/detection_loss‚1.3150256710384596 trainer/global_step‚9399ú•høæl¸‚(:á †ÛôÒöϤÌ disk.md0.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375' disk.nvme0c0n1.out‚42180.9853515625$ disk./.usageGB‚1170.921989440918( disk.nvme1c1n1.out‚13581.73974609375 disk.dm-1.in‚0$ memory_percent‚96.63717746734619( proc.memory.availableMB‚ 550.96484375" proc.memory.rssMB‚ 2789.3984375 network.sent‚ 1.2876162e+07% disk.nvme0n1.out‚42180.9853515625 disk.nvme1n1.in‚0 disk.dm-1.out‚ 7081.43359375 disk.rbd0.in‚0* proc.memory.percent‚17.025136947631836 disk.nvme1c1n1.in‚0 disk.md0.out‚ 127.48828125 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚48407.21826171875 cpu‚26.121015152519757 proc.cpu.threads‚17 network.recv‚ 1.318454e+06) disk./.usagePercent‚76.26914114580153 disk.nvme0n1.in‚ 4.3359375& disk.nvme1n1.out‚13581.73974609375K‚e듹‚(:ˆ †ÛôÒ¡çË gpu.0.gpu‚90.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚120.417' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.139 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ov&dº‚(:Ù •ÛôÒƒ¹øË network.sent‚ 1.2925629e+07 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 128.1328125 disk.dm-0.in‚ 4.3359375 disk.dm-1.out‚ 7082.03515625 disk.rbd0.out‚ 0.4697265625% proc.memory.availableMB‚ 550.59375 network.recv‚ 1.32267e+06 disk.nvme0c0n1.in‚ 4.3359375$ disk.nvme0n1.out‚42852.869140625 disk.nvme1n1.in‚0 disk.dm-1.in‚0! proc.memory.rssMB‚ 2789.703125* proc.memory.percent‚17.026996612548828 cpu‚26.119884155853995& disk.nvme0c0n1.out‚42852.869140625 disk.nvme0n1.in‚ 4.3359375( disk.nvme1c1n1.out‚13598.62841796875& disk.nvme1n1.out‚13598.62841796875# disk.dm-0.out‚49094.01806640625 disk.rbd0.in‚0$ memory_percent‚96.63944244384766 proc.cpu.threads‚17) disk./.usagePercent‚76.32950268900564$ disk./.usageGB‚1171.848690032959(ûäZœ»‚(:‘ •ÛôÒëÐÄÌ gpu.0.gpu‚41.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚178.84' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚59.61333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I5p2¼- _wandbruntime‚4891  _runtime‚4891p;Ž¿ trainer/global_step‚9449 # _timestamp‚1.78449141449453e+09 _runtime‚4891.069535518 _step‚763 & lr-AdamW/pg1‚0.0001883543865934379 & lr-AdamW/pg2‚0.0001883543865934379v+3hÊ¾Ä & lr-AdamW/pg2‚0.0001883543865934379 trainer/global_step‚9449 # _timestamp‚1.78449141449453e+09 _runtime‚4891.069535518 _step‚763 & lr-AdamW/pg1‚0.0001883543865934379û=Bé×2¿- _wandbruntime‚4891  _runtime‚4891 z¦¦è'ÀâO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159691776 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9342202913792 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 trainer/global_step‚9449 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚714982142298160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 _runtime‚ 4891.07058563 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9063928465408 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚722549103056384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169015658 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 135267328 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73550686 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95464972 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚731612690735616 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95464972 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚714982125258800 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73550686 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169025803 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9063587679232 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75024747 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73550688 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75024454 % _timestamp‚1.7844914144956229e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95475115 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚722549120574976 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚724029764180160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚731613049040384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚583279331753472 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚724029407217478 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95475115 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24956654 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260žà "ÁçO % _timestamp‚1.7844914144956229e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95475115 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚722549120574976 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚724029764180160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚731613049040384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚583279331753472 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚724029407217478 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95475115 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24956654 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159691776 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9342202913792 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 trainer/global_step‚9449 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚714982142298160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 _runtime‚ 4891.07058563 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9063928465408 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚722549103056384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169015658 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 135267328 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73550686 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95464972 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚731612690735616 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95464972 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚714982125258800 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73550686 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169025803 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9063587679232 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75024747 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73550688 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75024454üN¯xÔ2Â- _wandbruntime‚4891  _runtime‚4891ð£gó&ÃíM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚731688170727936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9064517590528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24959432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169043194 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73558190 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73558188 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚722623294825472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚722623312344064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚583339005264384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50072883 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95485004 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73558190 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚573995845480448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 _runtime‚4891.444481255 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9343292946944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169043194 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚722623312344064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95474847 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9064858383872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚715055568380640 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75032027 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚583339162851840 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169033035 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95474847 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133163008 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9048550105348 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169033035 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95485004 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50072881 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚573995869904896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚731688170727936 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚715055551341280 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 _step‚765×-Mø&ÄòM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169033035 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95474847 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133163008 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9048550105348 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169033035 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95485004 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50072881 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚573995869904896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚731688170727936 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚715055551341280 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 _step‚765 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚731688170727936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9064517590528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24959432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169043194 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73558190 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73558188 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚722623294825472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚722623312344064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚583339005264384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50072883 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95485004 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73558190 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚573995845480448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 _runtime‚4891.444481255 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9343292946944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169043194 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚722623312344064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95474847 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9064858383872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚715055568380640 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75032027 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚583339162851840 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚727747511369168 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73925786 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚576880785775104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9094064947008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚726258709029376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚726258691510784 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚735368807011840 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25082197 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚735368807011840 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚735369165316608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 53887G~¼Â}0784 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95970717 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25082488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚586271101729280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚727747154406486 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95960574 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169896505 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚718653429382800 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75404047 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9390291529728 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169886360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚726258709029376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9110456287232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73925786 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73925788 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50321852 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚9499 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9110456287232 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75404340 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚735369165316608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚586270944134656 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚576880810199552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9390158359552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9110115501056 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 % _timestamp‚1.7844914399956286e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚726258691510784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚735369165316608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚586270944134656 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚576880810199552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9390158359552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9110115501056 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 % _timestamp‚1.7844914399956286e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚726258691510784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚727747511369168 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73925786 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚576880785775104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9094064947008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚726258709029376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚726258691510784 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚735368807011840 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25082197 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚735368807011840 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚735369165316608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95970717 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25082488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚586271101729280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚727747154406486 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95960574 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169896505 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚718653429382800 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75404047 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9390291529728 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169886360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚726258709029376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9110456287232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73925786 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73925788 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50321852 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚9499 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9110456287232 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75404340 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75411631 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95980606 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169913896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚726332883279872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚718726872504640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25084988 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚576939662107648 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73933288 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚735443928692224 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95970449 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚586330933237248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9094993170356 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 trainer/global_step‚9499 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169903737 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚726332900798464 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50326933 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚726332883279872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9111386205696 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚727821508712202 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9111386205696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚718726855465280 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚576939686532096 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169913896 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9111045412352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9391115639296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚4916.945458887 _step‚769 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73933288 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚586330777746944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9391246705152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50326931 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9094653246922 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9111045412352 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 trainer/global_step‚9499 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169903737 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚726332900798464 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50326933 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚726332883279872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9111386205696 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚727821508712202 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9111386205696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚718726855465280 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚576939686532096 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169913896 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9111045412352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9391115639296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚4916.945458887 _step‚769 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73933288 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚586330777746944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9391246705152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50326931 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9094653246922 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9111045412352 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75411631 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95980606 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169913896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚726332883279872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚718726872504640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25084988 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚576939662107648 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73933288 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720™â~G2Ó- _wandbruntime‚4916  _runtime‚4916‘qõK Ô„ 3loss/SRqqt1/assignment_loss‚0.48177880552754965 3loss/SRqqt2/assignment_loss‚0.49371223825288457 2loss/FBt2/assignment_loss‚0.014746693541389199 1loss/SRqqt2/detection_loss‚0.9109330219609371 epoch‚11 trainer/global_step‚9499 /loss/FRt2/assignment_loss‚1.796808960099515 3loss/FBt1/assignment_loss‚9.946233323122354e-06 _step‚770 /loss/FRt1/detection_loss‚1.2978643383594581 /loss/FRt2/detection_loss‚0.5422082344823671 1loss/SRqqt1/detection_loss‚0.5855929153864115 0loss/FBt2/detection_loss‚0.29333866904626654 $loss/total_loss‚7.57702753435876 # _timestamp‚1.78449144037144e+09 0loss/FRt1/assignment_loss‚0.8146038800477982 /loss/FBt1/detection_loss‚0.3454298314208595 _runtime‚4916.946206669ÀN<*Õ‰ 3loss/SRqqt1/assignment_loss‚0.48177880552754965 3loss/SRqqt2/assignment_loss‚0.49371223825288457 2loss/FBt2/assignment_loss‚0.014746693541389199 1loss/SRqqt2/detection_loss‚0.9109330219609371 epoch‚11 trainer/global_step‚9499 /loss/FRt2/assignment_loss‚1.796808960099515 3loss/FBt1/assignment_loss‚9.946233323122354e-06 _step‚770 /loss/FRt1/detection_loss‚1.2978643383594581 /loss/FRt2/detection_loss‚0.5422082344823671 1loss/SRqqt1/detection_loss‚0.5855929153864115 0loss/FBt2/detection_loss‚0.29333866904626654 $loss/total_loss‚7.57702753435876 # _timestamp‚1.78449144037144e+09 0loss/FRt1/assignment_loss‚0.8146038800477982 /loss/FBt1/detection_loss‚0.3454298314208595 _runtime‚4916.946206669‚chï1žÖ‚(:“ ³ÛôÒÛ£áË gpu.0.gpu‚93.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚118.447' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.48233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê›/æ]ׂ(:Ò ³ÛôÒ“¼£Ì disk.nvme1c1n1.in‚0$ disk.nvme1n1.out‚13600.431640625 disk.dm-0.in‚ 4.3359375! disk.nvme0c0n1.out‚ 43352.3125 disk.nvme0n1.in‚ 4.3359375 disk.nvme0n1.out‚ 43352.3125' proc.memory.availableMB‚ 549.9296875 cpu‚26.117732717681957" disk.dm-1.out‚7082.35009765625 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625& disk.nvme1c1n1.out‚13600.431640625 disk.nvme1n1.in‚0$ memory_percent‚96.64349555969238) disk./.usagePercent‚76.39538457932913% disk./.usageGB‚1172.8601417541504 disk.md0.out‚ 128.85546875 disk.dm-1.in‚0! proc.memory.rssMB‚ 2790.015625) proc.memory.percent‚17.02890396118164 proc.cpu.threads‚17 network.sent‚ 1.3004516e+07 network.recv‚ 1.330046e+06 disk.md0.in‚0# disk.dm-0.out‚49593.36376953125 disk.nvme0c0n1.in‚ 4.3359375Ða—fØ‚(:Û ÂÛôÒÑüË) disk./.usagePercent‚76.50986351382167$ disk.nvme1c1n1.out‚ 13601.0859375( disk.nvme0c0n1.out‚45269.73095703125( proc.memory.availableMB‚ 549.89453125) proc.memory.percent‚17.02907085418701 cpu‚26.116823641482988 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375" disk.dm-1.out‚7082.49462890625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 4.3359375 network.recv‚ 1.334298e+06% disk./.usageGB‚1174.6176795959473 disk.md0.out‚ 129.21875# proc.memory.rssMB‚ 2790.04296875 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 13601.0859375" disk.dm-0.out‚51510.5224609375 disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375& disk.nvme0n1.out‚45269.73095703125$ memory_percent‚96.64371013641357 network.sent‚ 1.3053792e+07fǵ,žÙ‚(:“ ÂÛôÒùÁ÷Ë gpu.0.gpu‚90.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚239.788' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚79.92933333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Äàµæ2Ú- _wandbruntime‚4941  _runtime‚4941sŽç&ÆÛÀ $ _timestamp‚1.784491465327953e+09 _runtime‚4941.902959374 _step‚771 & lr-AdamW/pg1‚0.0001878080836373045 & lr-AdamW/pg2‚0.0001878080836373045 trainer/global_step‚9549áÇoËÜÅ & lr-AdamW/pg1‚0.0001878080836373045 & lr-AdamW/pg2‚0.0001878080836373045 trainer/global_step‚9549 $ _timestamp‚1.784491465327953e+09 _runtime‚4941.902959374 _step‚771ƒ?!ßü2Ý- _wandbruntime‚4941  _runtime‚4941Ï2|é'ÞãO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9156984109056 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚722324733506800 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50575900 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25207371 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚589262850934784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9438107715072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚731464901595494 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96466319 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9438224108032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74300886 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96466319 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75783539 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74300888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74300886 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50575902 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚4941.907465432 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚731465258558176 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170767207 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 140817408 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 116392960 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9140168088694 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚739125281592832 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 % _timestamp‚1.7844914653289554e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74300888 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚739124923288064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚579824602402304 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚722324750546160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 _step‚772 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25207637 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9140508012016 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚729968297483776 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚729968297483776 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚729968279965184 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9156643322880 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96456176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚589262710117376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96456176 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784ºÙ¡8î'ßèO VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9438107715072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚731464901595494 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96466319 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9438224108032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74300886 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96466319 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75783539 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74300888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74300886 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50575902 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚4941.907465432 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚731465258558176 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170767207 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 140817408 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 116392960 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9140168088694 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚739125281592832 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 % _timestamp‚1.7844914653289554e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74300888 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚739124923288064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚579824602402304 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚722324750546160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 _step‚772 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25207637 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9140508012016 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚729968297483776 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚729968297483776 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚729968279965184 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9156643322880 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96456176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚589262710117376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96456176 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9156984109056 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚722324733506800 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50575900 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25207371 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚589262850934784„ «‚Æ2à- _wandbruntime‚4942  _runtime‚4942j§é áëM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9141096311930 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9549 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚722398159589280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 140810240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚773 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74308388 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚731539255901210 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 J=DeviceStatsMonitor.on_train_batch_end/active.large_p•ô Tool.freed‚74308388 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170784598 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25210102 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75790807 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74308390 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚589322549906432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚731539612864004 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96466051 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 170774439 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚739200403280384 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170774439 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚589322690716672 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9439071171584 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚730042471734272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚739200044968448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚730042489252864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚579883478734848 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96466051 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚731539255901210 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74308388 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170784598 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25210102 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75790807 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74308390 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚589322549906432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚731539612864004 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96466051 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 170774439 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚739200403280384 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170774439 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚589322690716672 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9439071171584 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚730042471734272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚739200044968448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚730042489252864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚579883478734848 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96466051 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9549 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚722398159589280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 140810240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚773 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74308388 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚742881039564288 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50829950 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171637909 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74675988 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚735183005747184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚735182648784502 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚742881397869056 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9203511930880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76162912 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9203171144704 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚742881039564288 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96951778 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚582768419029504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50829952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96951778 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚296 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96961921 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9486252840960 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74675986 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚725996054670160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96961921 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚733677885938176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74675988 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9486123865088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚592254696294912 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚725996037630800 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9186611153702 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 _runtime‚ 4966.974508 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 _step‚776 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚733677868419584 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9203171144704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25332666 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9186951077024 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 171627764 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76162616 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚9599ó)Š#ïåO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚742881039564288 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96951778 OIDeviceStatsMonitor.on_train_batch_start/inactiv MÈ%e_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚582768419029504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50829952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96951778 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚296 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96961921 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9486252840960 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74675986 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚725996054670160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96961921 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚733677885938176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74675988 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9486123865088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚592254696294912 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚725996037630800 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9186611153702 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 _runtime‚ 4966.974508 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 _step‚776 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚733677868419584 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9203171144704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25332666 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9186951077024 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 171627764 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76162616 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚9599 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚582768443453952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚294 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚733677885938176 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9203511930880 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚592254542894592 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 % _timestamp‚1.7844914903994713e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚742881397869056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74675986 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25332960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚742881039564288 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50829950 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171637909 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74675988 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚735183005747184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚735182648784502 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚742881397869056 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9203511930880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76162912 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9203171144704 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087ˆ0/Ó2ð- _wandbruntime‚4967  _runtime‚4967¶üÏò&ñìM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50835033 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚582827319786496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9204441849344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171655300 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚733752077707264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25335145 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚582827295362048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 171645141 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚4967.345424846 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76170176 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9187539376938 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚742956519556608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚733752060188672 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171645141 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9487212101120 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚742956161244672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9204441849344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚592314380591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9487085229568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74683488 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74683490 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96961653 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚742956519556608 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚733752077707264 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚726069480752640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25335432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 ú1÷&òñM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚742956519556608 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚733752077707264 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚726069480752640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25335432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50835033 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚582827319786496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9204441849344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171655300 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚733752077707264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25335145 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚582827295362048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 171645141 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚4967.345424846 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76170176 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9187539376938 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚742956519556608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚733752060188672 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171645141 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9487212101120 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚742956161244672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9204441849344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚592314380591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9487085229568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74683488 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74683490 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96961653 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0‰mG¶™2ó- _wandbruntime‚4967  _runtime‚4967”MG ô† 2loss/SRqqt2/assignment_loss‚0.5981767218754941 2loss/FBt1/assignment_loss‚0.003366389920111877 /loss/FRt2/detection_loss‚0.5161572502302435 0loss/FRt1/assignment_loss‚0.9696590395407243 1loss/SRqqt2/detection_loss‚0.7725654908018299 %loss/total_loss‚8.322815588397042 epoch‚11 trainer/global_step‚9599 _runtime‚4967.346014057 /loss/FRt1/detection_loss‚1.3924860251881181 1loss/SRqqt1/detection_loss‚0.6052141995751299 _step‚778 2loss/SRqqt1/assignment_loss‚0.5085019371713421 3loss/FBt2/assignment_loss‚0.0010318165557076436 0loss/FBt1/detection_loss‚0.45948459309831596 /loss/FBt2/detection_loss‚0.6821223290778858 % _timestamp‚1.7844914907713444e+09 0loss/FRt2/assignment_loss‚1.8140497953621382OÇZ=õ‹ 0loss/FRt1/assignment_loss‚0.9696590395407243 1loss/SRqqt2/detection_loss‚0.7725654908018299 %loss/total_loss‚8.322815588397042 epoch‚11 trainer/global_step‚9599 _runtime‚4967.346014057 /loss/FRt1/detection_loss‚1.3924860251881181 1loss/SRqqt1/detection_loss‚0.6052141995751299 _step‚778 2loss/SRqqt1/assignment_loss‚0.5085019371713421 3loss/FBt2/assignment_loss‚0.0010318165557076436 0loss/FBt1/detection_loss‚0.45948459309831596 /loss/FBt2/detection_loss‚0.6821223290778858 % _timestamp‚1.7844914907713444e+09 0loss/FRt2/assignment_loss‚1.8140497953621382 2loss/SRqqt2/assignment_loss‚0.5981767218754941 2loss/FBt1/assignment_loss‚0.003366389920111877 /loss/FRt2/detection_loss‚0.5161572502302435Šm>´"mö‚(:â ïÛôÒ¤ƒžÌ disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 cpu‚26.117520134308723 disk.dm-0.out‚52391.19921875 disk.dm-1.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2834.71484375 network.sent‚ 1.3139539e+07) disk./.usagePercent‚76.50684281526064( disk.nvme1c1n1.out‚14443.31298828125 disk.nvme1n1.in‚0& disk.nvme1n1.out‚14443.31298828125( disk.nvme0c0n1.out‚45848.58154296875 proc.cpu.threads‚17 network.recv‚ 1.345151e+06$ disk./.usageGB‚1174.571304321289 disk.rbd0.out‚ 0.4697265625& disk.nvme0n1.out‚45848.58154296875$ memory_percent‚96.91660404205322( proc.memory.availableMB‚ 505.18359375) proc.memory.percent‚17.30172634124756 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 130.30078125 disk.dm-0.in‚ 4.3359375! disk.dm-1.out‚7620.5791015625Z雓÷‚(:ˆ ïÛôÒ¿ÂÐÌ gpu.0.gpu‚76.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚52.0 gpu.0.powerWatts‚118.155' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.385 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vê¥â2ø- _wandbruntime‚4992  _runtime‚4992€„)BOù·{ 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912552833557129 9"Purity/0FRt1ï‡îæn=SRqqt1FBt/event_purity‚0.6275806427001953 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9988662004470825 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.323566198348999 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9804894328117371 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06097731739282608 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9991687536239624 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9277232885360718 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6906878352165222 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3249704837799072 +particle/sensitivity‚0.6485022306442261 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4996018707752228 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.754963755607605 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9821428656578064 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 particle/accuracy_2_of_3‚1 /particle/accuracy_4_of_4‚0.9538451433181763 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9750000238418579 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.266638845205307 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7124306559562683 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9957055449485779 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05530375614762306 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03436247631907463 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30264008045196533 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9905115962028503 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.031243303790688515 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 *jet/accuracy_1_of_2‚0.8101272583007812 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 _runtime‚4992.146588699 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15729506313800812 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6592949628829956 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6604097485542297 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8706141114234924 epoch‚11 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.30554452538490295 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6345067024230957 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9909163117408752 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3145121932029724 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_1_of_4‚0.9745267033576965 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7890170216560364 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23696205019950867 % _timestamp‚1.7844915155711555e+09 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.11711254715919495 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7validation_average_jet_accuracy‚0.40382620692253113 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.05000000074505806 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.043217938393354416 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.31239500641822815 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6548077464103699 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8100000023841858 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2847222089767456 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7632790803909302 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7685281038284302 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2551054358482361 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5751553773880005 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.24722063541412354 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7552231550216675 /particle/accuracy_3_of_4‚0.9973545074462891 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03456820920109749 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2850128710269928 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_3_of_4‚0.09049476683139801 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6145833134651184 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9986110925674438 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_3‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9059023261070251 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.516730010509491 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1412409543991089 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4888193905353546 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6604097485542297 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_2_of_4‚1 jet/accuracy_1_of_5‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9095184803009033 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8540598750114441 _step‚779 *jet/accuracy_2_of_4‚0.8109366297721863 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.031607430428266525 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.397019624710083 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_1‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7615473866462708 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0546366423368454 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7610856294631958 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8100000023841858 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9967120289802551 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.6037400364875793 particle/accuracy_2_of_2‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9967724680900574 particle/accuracy_1_of_5‚1 trainer/global_step‚9611 (loss/val_total_loss‚8.11830775560497 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.16801491379737854 +jet/accuracy_4_of_4‚0.01666666567325592 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6608701944351196 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.42424246668815613 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9970436096191406 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7005427479743958 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39236682653427124 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.023426871746778488 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.859833836555481 +jet/accuracy_3_of_3‚0.07792095839977264 'particle/f_score‚0.6552256345748901 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6711100339889526 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6470103859901428 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.859833836555481 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9970735907554626 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23930220305919647 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9213560223579407 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6576189994812012 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9964585900306702 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 (particle/accuracy‚0.8534730076789856 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7016101479530334 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.27004534006118774 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7697491645812988 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.05000000074505806 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23258933424949646 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14360080659389496 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8120346069335938 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3251759111881256 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.696959912776947 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3675788640975952 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3244021534919739 jet/accuracy_4_of_5‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22832366824150085 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6700555682182312 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5183261036872864 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12083333730697632 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7150378823280334 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23742589354515076 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.37150976061820984 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7543717622756958 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023426871746778488 particle/accuracy_3_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9533917307853699 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8618420958518982 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8828428387641907 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_3_of_3‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5039916038513184 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.25894615054130554 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5832917094230652 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7835586071014404 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22386875748634338 , Purity/2FRt*SRqqt1FBt/FRt_purity‚0.53125 particle/accuracy_1_of_4‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8618420958518982 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8885067701339722 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9964616894721985 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7442631721496582 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6458937525749207 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7114806771278381 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3145121932029724 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6625809669494629 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8116836547851562 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.10521351546049118 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7888889312744141 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6674092411994934 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0357142873108387 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.33718788623809814 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7631040215492249 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12782926857471466 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4682350754737854 particle/accuracy_1_of_2‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8572649955749512 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0357142873108387 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.767458975315094 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9902447462081909 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7030904293060303 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22124716639518738 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4749999940395355 +jet/accuracy_1_of_1‚0.36683934926986694 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740737318992615 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.99801588058471688Sd™Â=ú¼{ =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15729506313800812 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6592949628829956 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6604097485542297 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8706141114234924 epoch‚11 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.30554452538490295 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6345067024230957 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9909163117408752 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3145121932029724 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_1_of_4‚0.9745267033576965 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7890170216560364 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23696205019950867 % _timestamp‚1.7844915155711555e+09 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.11711254715919495 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7validation_average_jet_accuracy‚0.40382620692253113 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.05000000074505806 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.043217938393354416 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.31239500641822815 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6548077464103699 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8100000023841858 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2847222089767456 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7632790803909302 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7685281038284302 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2551054358482361 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5751553773880005 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.24722063541412354 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7552231550216675 /particle/accuracy_3_of_4‚0.9973545074462891 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03456820920109749 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2850128710269928 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_3_of_4‚0.09049476683139801 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6145833134651184 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9986110925674438 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_3‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9059023261070251 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.516730010509491 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1412409543991089 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4888193905353546 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6604097485542297 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_2_of_4‚1 jet/accuracy_1_of_5‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9095184803009033 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8540598750114441 _step‚779 *jet/accuracy_2_of_4‚0.8109366297721863 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.031607430428266525 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.397019624710083 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_1‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7615473866462708 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0546366423368454 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7610856294631958 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8100000023841858 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9967120289802551 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.6037400364875793 particle/accuracy_2_of_2‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9967724680900574 particle/accuracy_1_of_5‚1 trainer/global_step‚9611 (loss/val_total_loss‚8.11830775560497 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.16801491379737854 +jet/accuracy_4_of_4‚0.01666666567325592 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6608701944351196 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.42424246668815613 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9970436096191406 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7005427479743958 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39236682653427124 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.023426871746778488 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.859833836555481 +jet/accuracy_3_of_3‚0.07792095839977264 'particle/f_score‚0.6552256345748901 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6711100339889526 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6470103859901428 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.859833836555481 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9970735907554626 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23930220305919647 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9213560223579407 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6576189994812012 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9964585900306702 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 (particle/accuracy‚0.8534730076789856 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7016101479530334 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.27004534006118774 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7697491645812988 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.05000000074505806 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.23258933424949646 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14360080659389496 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8120346069335938 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3251759111881256 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.696959912776947 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3675788640975952 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3244021534919739 jet/accuracy_4_of_5‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22832366824150085 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6700555682182312 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5183261036872864 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12083333730697632 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7150378823280334 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23742589354515076 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.37150976061820984 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7543717622756958 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023426871746778488 particle/accuracy_3_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9533917307853699 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8618420958518982 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8828428387641907 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_3_of_3‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5039916038513184 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.25894615054130554 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5832917094230652 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7835586071014404 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22386875748634338 , Purity/2FRt*SRqqt1FBt/FRt_purity‚0.53125 particle/accuracy_1_of_4‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8618420958518982 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8885067701339722 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9964616894721985 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7442631721496582 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6458937525749207 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7114806771278381 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3145121932029724 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6625809669494629 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8116836547851562 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.10521351546049118 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7888889312744141 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6674092411994934 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0357142873108387 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.33718788623809814 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7631040215492249 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12782926857471466 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4682350754737854 particle/accuracy_1_of_2‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8572649955749512 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0357142873108387 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.767458975315094 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9902447462081909 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7030904293060303 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22124716639518738 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4749999940395355 +jet/accuracy_1_of_1‚0.36683934926986694 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740737318992615 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912552833557129 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6275806427001953 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9988662004470825 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.323566198348999 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9804894328117371 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06097731739282608 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9991687536239624 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9277232885360718 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6906878352165222 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3249704837799072 +particle/sensitivity‚0.6485022306442261 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4996018707752228 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.754963755607605 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9821428656578064 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 particle/accuracy_2_of_3‚1 /particle/accuracy_4_of_4‚0.9538451433181763 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9750000238418579 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.266638845205307 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7124306559562683 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9957055449485779 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05530375614762306 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03436247631907463 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30264008045196533 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9905115962028503 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.031243303790688515 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 *jet/accuracy_1_of_2‚0.8101272583007812 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 _runtime‚4992.146588699‹ 4•ndû‚(:Ù þÛôÒžó•Ì% disk.nvme0n1.out‚45849.5751953125 disk.md0.in‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 4.3359375& disk.nvme1c1n1.out‚14689.376953125( proc.memory.availableMB‚ 737.14453125# proc.memory.rssMB‚ 2839.79296875 cpu‚26.1235076049162 disk.md0.out‚ 130.7578125! disk.dm-1.out‚7866.0166015625 disk.nvme1c1n1.in‚0$ memory_percent‚95.50082683563232) proc.memory.percent‚17.33272075653076 proc.cpu.threads‚17 network.sent‚ 1.3200954e+07) disk./.usagePercent‚76.50684281526064 disk.nvme1n1.in‚0 disk.dm-0.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 network.recv‚ 1.349336e+06$ disk./.usageGB‚1174.571304321289$ disk.nvme1n1.out‚14689.376953125 disk.dm-0.out‚52391.84765625 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625' disk.nvme0c0n1.out‚45849.5751953125*,—}Pü‚(:“ þÛôÒÆ“ÌÌ gpu.0.gpu‚92.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚252.751' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.25033333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚÷øg‡N1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØòŒijý‚(:ß ÜôÒ¥ßË disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0$ memory_percent‚95.50256729125977# proc.memory.rssMB‚ 2839.79296875 proc.cpu.threads‚17) disk./.usagePercent‚76.50684306373465" disk.dm-1.out‚7866.16259765625 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚14690.3603515625 cpu‚26.125233524095737 network.sent‚ 1.3246971e+07 disk.md0.out‚ 131.4453125! disk.dm-0.out‚52392.916015625 disk.nvme0c0n1.in‚ 4.3359375& proc.memory.availableMB‚ 736.859375) proc.memory.percent‚17.33272075653076% disk./.usageGB‚1174.5713081359863( disk.nvme0c0n1.out‚45851.26220703125& disk.nvme0n1.out‚45851.26220703125' disk.nvme1c1n1.out‚14690.3603515625 disk.nvme1n1.in‚0 network.recv‚ 1.35276e+06+ÊÚ/‘þ‚(:† ÜôÒ­ÀÕË gpu.0.gpu‚56.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚252.57' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚84.19 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 \‘—2ÿ- _wandbruntime‚5012  _runtime‚5012<—ðÇ€Á & lr-AdamW/pg1‚0.0001872565666172219 & lr-AdamW/pg2‚0.0001872565666172219 trainer/global_step‚9649 % _timestamp‚1.7844915357128873e+09 _runtime‚5012.288392861 _step‚780þà8ÌÆ trainer/global_step‚9649 % _timestamp‚1.7844915357128873e+09 _runtime‚5012.288392861 _step‚780 & lr-AdamW/pg1‚0.0001872565666172219 & lr-AdamW/pg2‚0.0001872565666172219Œj7$2‚- _wandbruntime‚5012  _runtime‚5012á /é'ƒãO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚748609020853760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75292252 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚748608662548992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚748609020853760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97621093 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚5012.288581011 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 _step‚781 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9274555807744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51270482 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9274215021568 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9257544222180 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497472 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9274215021568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51270480 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚587305121748480 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚596864159743488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25562974 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚596864004246016 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9558882497536 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚739334465046016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚739334465046016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25563251 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚740842314103502 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚731584429958000 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚731584412918640 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75292250 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚739334447527424 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 % _timestamp‚1.7844915357136185e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 172903200 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76833733 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97610950 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97610950 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚740841957140820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚587305146172928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97621093 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 172913345 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837w§›†î'„èO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚587305146172928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97621093 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 172913345 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚748609020853760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75292252 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚748608662548992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚748609020853760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97621093 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚5012.288581011 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 _step‚781 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9274555807744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51270482 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9274215021568 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9257544222180 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497472 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9274215021568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51270480 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚587305121748480 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚596864159743488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25562974 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚596864004246016 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9558882497536 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚739334465046016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚739334465046016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25563251 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚740842314103502 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚731584429958000 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚731584412918640 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75292250 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚739334447527424 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 % _timestamp‚1.7844915357136185e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 172903200 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76833733 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97610950 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97610950 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚740841957140820‘ö>2…- _wandbruntime‚5012  _runtime‚5012ý«‡<ò&†ìM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25565785 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 trainer/global_step‚9649 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9275485726208 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51275563 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75299754 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75299752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚739408656815104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25565509 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚740916668409330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 _runtime‚5012.658493033 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚740916311446536 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9258812368850 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 172920577 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9559844123136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 172920577 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97630982 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.784491536083279e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172930736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97620825 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 _step‚782 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 172930736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75299752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97630982 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9559970994688 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51275561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚596923842204160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚748684142541312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9258472445416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9275485726208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75299754 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76841348 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚748684142541312 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚739408656815104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚596923993500160 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚731657856040480 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97620825 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9275144932864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76841070 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0Ñåcq‡ñM UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚596923993500160 P@DeviceStatsMonitor.on_train_batchm«æ †%_end/reserved_bytes.all.current‚ 14407434240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚731657856040480 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97620825 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9275144932864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76841070 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25565785 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 trainer/global_step‚9649 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9275485726208 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51275563 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75299754 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75299752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚739408656815104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25565509 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚740916668409330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 _runtime‚5012.658493033 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚740916311446536 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9258812368850 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 172920577 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9559844123136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 172920577 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97630982 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.784491536083279e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172930736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97620825 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 _step‚782 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 172930736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75299752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97630982 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9559970994688 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51275561 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚596923842204160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚748684142541312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9258472445416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9275485726208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75299754 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76841348 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚748684142541312 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚739408656815104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344Ž3@2ˆ- _wandbruntime‚5012  _runtime‚5012.`ž5 ‰† /loss/FRt2/assignment_loss‚1.841564141087613 0loss/FBt1/detection_loss‚0.37980780447148443 0loss/FBt2/detection_loss‚0.48548805540485773 trainer/global_step‚9649 0loss/FRt1/assignment_loss‚0.8060090382551325 /loss/FRt1/detection_loss‚1.4724386557555307 % _timestamp‚1.7844915360848427e+09 _runtime‚5012.659785655 2loss/SRqqt1/assignment_loss‚0.4995472600800609 1loss/SRqqt2/assignment_loss‚0.524486055685361 2loss/FBt1/assignment_loss‚0.000389263578209806 1loss/SRqqt1/detection_loss‚0.5968722052925399 epoch‚12 _step‚783 4loss/FBt2/assignment_loss‚0.00025828391533577744 /loss/FRt2/detection_loss‚0.5490299933882961 1loss/SRqqt2/detection_loss‚0.8756866231211461 %loss/total_loss‚8.031577380035568iÔÍŠ‹ _runtime‚5012.659785655 2loss/SRqqt1/assignment_loss‚0.4995472600800609 1loss/SRqqt2/assignment_loss‚0.524486055685361 2loss/FBt1/assignment_loss‚0.000389263578209806 1loss/SRqqt1/detection_loss‚0.5968722052925399 epoch‚12 _step‚783 4loss/FBt2/assignment_loss‚0.00025828391533577744 /loss/FRt2/detection_loss‚0.5490299933882961 1loss/SRqqt2/detection_loss‚0.8756866231211461 %loss/total_loss‚8.031577380035568 /loss/FRt2/assignment_loss‚1.841564141087613 0loss/FBt1/detection_loss‚0.37980780447148443 0loss/FBt2/detection_loss‚0.48548805540485773 trainer/global_step‚9649 0loss/FRt1/assignment_loss‚0.8060090382551325 /loss/FRt1/detection_loss‚1.4724386557555307 % _timestamp‚1.7844915360848427e+09wJlk‹‚(:à œÜôÒ¹íÇË& disk.nvme1c1n1.out‚14936.318359375 disk.nvme1n1.in‚0 network.sent‚ 1.3249893e+07) disk./.usagePercent‚76.50684430610471 disk.rbd0.in‚0& disk.nvme0n1.out‚45852.08349609375 disk.nvme1c1n1.in‚0! disk.dm-0.out‚52393.505859375 disk.rbd0.out‚ 0.4697265625( disk.nvme0c0n1.out‚45852.08349609375 disk.dm-1.in‚0$ memory_percent‚95.50127983093262' proc.memory.availableMB‚ 737.0703125) proc.memory.percent‚17.33272075653076 cpu‚26.127293773261915$ disk.nvme1n1.out‚14936.318359375 network.recv‚ 1.356164e+06% disk./.usageGB‚1174.5713272094727 disk.nvme0c0n1.in‚ 4.3359375 disk.md0.out‚ 131.73828125 disk.dm-0.in‚ 4.3359375" disk.dm-1.out‚8111.70166015625# proc.memory.rssMB‚ 2839.79296875 disk.nvme0n1.in‚ 4.3359375 disk.md0.in‚0 proc.cpu.threads‚17¸c“Œ‚(:ˆ œÜôÒûܦÌ gpu.0.gpu‚70.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚52.0 gpu.0.powerWatts‚118.155' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.385 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àÜŠ2- _wandbruntime‚5037  _runtime‚5037ÈÓŽêÉŽÃ _runtime‚5037.906862425 _step‚784 ' lr-AdamW/pg1‚0.00018697886386224885 ' lr-AdamW/pg2‚0.00018697886386224885 trainer/global_step‚9699 % _timestamp‚1.7844915613318982e+09Z‹XóÎÈ _runtime‚5037.906862425 _step‚784 ' lr-AdamW/pg1‚0.00018697886386224885 ' lr-AdamW/pg2‚0.00018697886386224885 trainer/global_step‚9699 % _timestamp‚1.7844915613318982e+09ôyô2- _wandbruntime‚5037  _runtime‚5037&\Îé'‘ãO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9320742843392 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9606985576448 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173784047 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚590248962800128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚744559704329828 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9304327210510 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75667350 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚752364778825216 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚752365137129984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51524530 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚590248938375680 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9320742843392 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚599855792879104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98106552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚743044053500416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75667352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75667352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚743044035981824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9321083629568 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚735255717042640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚743044053500416 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98116695 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51524532 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚785 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚743044035981824 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25689119 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9303987287188 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75667350 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚752364778825216 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77213651 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 % _timestamp‚1.7844915613328497e+09 _runtime‚5037.911425122 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25688839 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0Ñç?oõ$’èO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚785 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚743044035981824 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25689119 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9303987287188 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75667350 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚752364778825216 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77213651 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 % _timestamp‚1.7844915613328497e+09 _runtime‚5037.911425122 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25688839 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9320742843392 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9606985576448 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173784047 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚590248962800128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚744559704329828 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9304327210510 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75667350 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚752364778825216 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚752365137129984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51524530 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚590248938375680 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9320742843392 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚599855792879104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98106552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚743044053500416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75667352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75667352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚743044035981824 [IDeviceStatsMonitor.on_train_batch +›ù_start/active_bytes.small_pool.allocated‚ 9321083629568 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚735255717042640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚743044053500416 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98116695 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51524532 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0‘ÛîÀR2“- _wandbruntime‚5038  _runtime‚5038ÂzÉÅó&”íM H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173791279 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98116427 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚743118227750912 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚599915782104576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9322013548032 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75674854 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98116427 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51529613 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚752439900505600 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚743118245269504 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25691294 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 % _timestamp‚1.7844915617034385e+09 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚744634058635544 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚735329160164480 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77220905 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚743118245269504 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75674852 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9607942971904 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9321672754688 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚752440258817536 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173801438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚590307839132672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚743118227750912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75674852 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77221179 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚599915628711424 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚735329143125120 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9607814003200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173801438 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚744634415598338 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51529611 _step‚786 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98126584 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25691566 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚590307814708224 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9305255433858 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9304915510424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 trainer/global_step‚9699 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9322013548032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _runtime‚5038.278545164 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9321672754688 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98126584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173801438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚590307839132672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚743118227750912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75674852 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77221179 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚599915628711424 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚735329143125120 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9607814003200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173801438 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚744634415598338 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51529611 _step‚786 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98126584 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25691566 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚590307814708224 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9305255433858 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9304915510424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 trainer/global_step‚9699 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9322013548032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _runtime‚5038.278545164 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9321672754688 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98126584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77220905 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚743118245269504 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75674852 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9607942971904 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720’Gwk2–- _wandbruntime‚5038  _runtime‚5038ãÖòÚ —† 2loss/FBt1/assignment_loss‚0.010251410786970238 4loss/FBt2/assignment_loss‚0.00031344117591148603 /loss/FBt2/detection_loss‚0.5302723422646523 epoch‚12 % _timestamp‚1.7844915617044716e+09 2loss/SRqqt2/assignment_loss‚0.5083101346877811 _runtime‚5038.279192835 _step‚787 2loss/SRqqt1/assignment_loss‚0.5433842877868762 /loss/FRt2/detection_loss‚0.5163293814877081 1loss/SRqqt1/detection_loss‚0.5894087601085783 0loss/FRt1/assignment_loss‚0.8054438862274951 .loss/FRt1/detection_loss‚1.296379076201454 1loss/SRqqt2/detection_loss‚0.9365378950780723 0loss/FBt1/detection_loss‚0.31521584048795587 %loss/total_loss‚7.841011900110068 trainer/global_step‚9699 0loss/FRt2/assignment_loss‚1.78916544381661319áñΘ‹ 2loss/SRqqt1/assignment_loss‚0.5433842877868762 /loss/FRt2/detection_loss‚0.5163293814877081 1loss/SRqqt1/detection_loss‚0.5894087601085783 0loss/FRt1/assignment_loss‚0.8054438862274951 .loss/FRt1/detection_loss‚1.296379076201454 1loss/SRqqt2/detection_loss‚0.9365378950780723 0loss/FBt1/detection_loss‚0.31521584048795587 %loss/total_loss‚7.841011900110068 trainer/global_step‚9699 0loss/FRt2/assignment_loss‚1.7891654438166131 2loss/FBt1/assignment_loss‚0.010251410786970238 4loss/FBt2/assignment_loss‚0.00031344117591148603 /loss/FBt2/detection_loss‚0.5302723422646523 epoch‚12 % _timestamp‚1.7844915617044716e+09 2loss/SRqqt2/assignment_loss‚0.5083101346877811 _runtime‚5038.279192835 _step‚787“êáÒo™‚(:ä «ÜôÒþÂóË disk.nvme1c1n1.in‚0# disk.dm-0.out‚52395.50634765625! disk.dm-1.out‚8111.9462890625 disk.rbd0.out‚ 0.4697265625( proc.memory.availableMB‚ 736.81640625 cpu‚26.129484535524465 network.sent‚ 1.3325072e+07 disk.rbd0.in‚0# proc.memory.rssMB‚ 2839.79296875 disk.nvme0n1.in‚ 4.3359375( disk.nvme1c1n1.out‚14937.12646484375& disk.nvme1n1.out‚14937.12646484375 disk.md0.in‚0 disk.md0.out‚ 131.99609375$ memory_percent‚95.50282955169678) proc.memory.percent‚17.33272075653076 proc.cpu.threads‚17 network.recv‚ 1.360409e+06) disk./.usagePercent‚76.50682641597587% disk./.usageGB‚1174.5710525512695 disk.nvme0c0n1.in‚ 4.3359375' disk.nvme0c0n1.out‚45854.1083984375 disk.nvme1n1.in‚0 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0% disk.nvme0n1.out‚45854.1083984375„ ˜kš‚(:’ «ÜôÒܦÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚52.0 gpu.0.powerWatts‚117.753' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.251000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Èdjäp›‚(:å ºÜôÒåöË* proc.memory.percent‚17.333483695983887 cpu‚26.131517844992572 network.sent‚ 1.3376428e+07) disk./.usagePercent‚76.51053315128338 disk.nvme0n1.in‚ 4.3359375 disk.rbd0.out‚ 0.4697265625$ memory_percent‚95.50225734710693# proc.memory.rssMB‚ 2839.91796875 proc.cpu.threads‚17% disk.nvme0n1.out‚45854.8544921875 disk.nvme1n1.in‚0 disk.md0.out‚ 132.25390625 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚52396.05712890625! disk.dm-1.out‚8112.0517578125' disk.nvme0c0n1.out‚45854.8544921875( proc.memory.availableMB‚ 736.91015625 network.recv‚ 1.364423e+06% disk./.usageGB‚1174.6279602050781 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚14937.60693359375& disk.nvme1n1.out‚14937.60693359375 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375"±ážœ‚(:“ ºÜôÒ˜³¥Ì gpu.0.gpu‚79.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚181.333' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚60.44433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½\á32- _wandbruntime‚5063  _runtime‚5063ÝÕFȞ _step‚788 ' lr-AdamW/pg1‚0.00018669987086770667 ' lr-AdamW/pg2‚0.00018669987086770667 trainer/global_step‚9749 $ _timestamp‚1.784491586621896e+09 _runtime‚5063.196852704@BJ‰ÍŸÇ ' lr-AdamW/pg2‚0.00018669987086770667 trainer/global_step‚9749 $ _timestamp‚1.784491586621896e+09 _runtime‚5063.196852704 _step‚788 ' lr-AdamW/pg1‚0.00018669987086770667”ùÊ?2 - _wandbruntime‚5063  _runtime‚5063ÇKÊ¡âO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚748277808481518 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚756121253406208 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚746753641954816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚756121253406208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51778580 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9367270665216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 _step‚789 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9367611451392 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76042450 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76042450 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25815361 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9654922201600 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9367270665216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚738927038206000 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚738927021166640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚602847546131456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9654791128576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 174654749 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚746753641954816 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚602847701628928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497472 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 174644604 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚756120895101440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76042452 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77593664 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76042452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98612297 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9367611451392 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚593192755002880 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚748277451518836 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77593943 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚756120895101440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25815084 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784491586622591e+09 _runtime‚5063.197453925 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98602154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9350430352196 trainer/global_step‚9749 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚746753624436224 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚746753624436224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51778582 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9350770275518 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚593192779427328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98602154 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98612297 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9367611451392 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚593192755002880 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚748277451518836 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77593943 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚756120895101440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25815084 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784491586622591e+09 _runtime‚5063.197453925 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98602154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9350430352196 trainer/global_step‚9749 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚746753624436224 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚746753624436224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51778582 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9350770275518 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚593192779427328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98602154 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76042450 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76042450 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25815361 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9654922201600•½…u2£- _wandbruntime‚5063  _runtime‚5063£j{ó&¤íM H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 174661981 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚739000464288480 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚756196016781824 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 174661981 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9368541369856 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844915873135414e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9351358575432 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚593251655759872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9655876336640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98622186 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚756196375093760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚739000447249120 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98622186 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77601268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚748351805824552 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9368541369856 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76049954 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9368200576512 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9655745270784 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76049954 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77601554 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚790 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚746827816205312 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚593251631335424 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 174672140 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51783663 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25817607 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚746827833723904 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚748352162787346‘Ç÷xû$¥òM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚748351805824552 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9368541369856 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76049954 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9368200576512 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9655745270784 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76049954 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77601554 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚790 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚746827816205312 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚593251631335424 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 174672140 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51783663 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25817607 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚746827833723904 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚748352162787346 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 174661981 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚739000464288480 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚756196016781824 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 174661981 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9368541369856 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844915873135414e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9351358575432 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚593251655759872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9655876336640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98622186 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚756196375093760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚739000447249120 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.smaÂt­—ýll_pool.allocated‚ 545259520 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98622186 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77601268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260–$.ª2¦- _wandbruntime‚5063  _runtime‚5063ö] §… _runtime‚5063.889835551 _step‚791 0loss/FRt1/assignment_loss‚0.7629258218875601 0loss/FRt2/assignment_loss‚1.8485536042914499 3loss/FBt1/assignment_loss‚2.133059786918972e-05 /loss/FRt1/detection_loss‚1.3303358478506273 1loss/SRqqt2/detection_loss‚0.8518534776444237 epoch‚12 2loss/SRqqt1/assignment_loss‚0.4908063386926008 1loss/SRqqt2/assignment_loss‚0.513285892981318 0loss/FBt1/detection_loss‚0.46987467109901254 $loss/total_loss‚8.30215114918364 trainer/global_step‚9749 % _timestamp‚1.7844915873146682e+09 /loss/FRt2/detection_loss‚0.5356723979186662 /loss/FBt2/detection_loss‚0.9409067347379667 3loss/FBt2/assignment_loss‚0.0010145390489079936 1loss/SRqqt1/detection_loss‚0.5569004924332371þ˜­&¨Š 0loss/FRt1/assignment_loss‚0.7629258218875601 0loss/FRt2/assignment_loss‚1.8485536042914499 3loss/FBt1/assignment_loss‚2.133059786918972e-05 /loss/FRt1/detection_loss‚1.3303358478506273 1loss/SRqqt2/detection_loss‚0.8518534776444237 epoch‚12 2loss/SRqqt1/assignment_loss‚0.4908063386926008 1loss/SRqqt2/assignment_loss‚0.513285892981318 0loss/FBt1/detection_loss‚0.46987467109901254 $loss/total_loss‚8.30215114918364 trainer/global_step‚9749 % _timestamp‚1.7844915873146682e+09 /loss/FRt2/detection_loss‚0.5356723979186662 /loss/FBt2/detection_loss‚0.9409067347379667 3loss/FBt2/assignment_loss‚0.0010145390489079936 1loss/SRqqt1/detection_loss‚0.5569004924332371 _runtime‚5063.889835551 _step‚791—†"Øgg©‚(:Ü ÉÜôÒ¦ñ·Ë network.sent‚ 1.3379392e+07) disk./.usagePercent‚76.45376553618544 disk.md0.in‚0! disk.dm-1.out‚8112.3408203125 disk.nvme1c1n1.in‚0$ memory_percent‚95.50414085388184 cpu‚26.133446263332218$ disk./.usageGB‚1173.756435394287 disk.md0.out‚ 132.62890625 disk.dm-0.in‚ 4.3359375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚45886.60498046875 disk.nvme1n1.in‚0 network.recv‚ 1.367647e+06# disk.dm-0.out‚52427.71142578125 disk.dm-1.in‚0$ disk.nvme1c1n1.out‚ 14938.6171875" disk.nvme1n1.out‚ 14938.6171875' proc.memory.availableMB‚ 736.6015625# proc.memory.rssMB‚ 2839.98046875) proc.memory.percent‚17.33386516571045 disk.rbd0.out‚ 0.4697265625( disk.nvme0c0n1.out‚45886.60498046875 proc.cpu.threads‚17ÞµÕª‚(:’ ÉÜôÒ«®œÌ gpu.0.gpu‚89.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚52.0 gpu.0.powerWatts‚118.72' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.57333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s6n«‚(:ã ØÜôÒô¼ïË disk.md0.in‚0# disk.dm-0.out‚52428.75830078125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625' disk.nvme0c0n1.out‚45887.9189453125* proc.memory.percent‚17.334628105163574 proc.cpu.threads‚17 disk.md0.out‚ 132.98828125 network.sent‚ 1.34436e+07 network.recv‚ 1.372024e+06) disk./.usagePercent‚76.45378168699621% disk./.usageGB‚1173.7566833496094% disk.nvme0n1.out‚45887.9189453125 disk.nvme1c1n1.in‚0( proc.memory.availableMB‚ 736.66015625 disk.nvme1n1.in‚0& disk.nvme1n1.out‚14939.24755859375 disk.dm-0.in‚ 4.3359375! disk.dm-1.out‚8112.4462890625 disk.nvme0c0n1.in‚ 4.3359375$ memory_percent‚95.50378322601318# proc.memory.rssMB‚ 2840.10546875 disk.nvme0n1.in‚ 4.3359375( disk.nvme1c1n1.out‚14939.24755859375 cpu‚26.135500506838753RNËž¬‚(:“ ØÜôÒ«Â¥Ì gpu.0.gpu‚61.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚262.951' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.65033333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ]¼©2­- _wandbruntime‚5088  _runtime‚5088?†õ:ɮà _step‚792 ' lr-AdamW/pg1‚0.00018641959210223444 ' lr-AdamW/pg2‚0.00018641959210223444 trainer/global_step‚9799 % _timestamp‚1.7844916122639196e+09 _runtime‚5088.838892722!0ÚäÎ¯È _step‚792 ' lr-AdamW/pg1‚0.00018641959210223444 ' lr-AdamW/pg2‚0.00018641959210223444 trainer/global_step‚9799 % _timestamp‚1.7844916122639196e+09 _runtime‚5088.838892722˜JuOO2°- _wandbruntime‚5088  _runtime‚5088Ni^é'±ãO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9702887163392 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚750463212890624 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9414139273216 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76417550 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52032632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25940505 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚596136596054528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚750463230409216 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99097756 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚759877011377664 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99107899 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚751995198707844 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175525451 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚759877011377664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9414139273216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚742598325290640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9397213340526 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76417550 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99107899 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚759877369682432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚750463230409216 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52032630 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚596136571630080 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚605839329817600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9413798487040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚9799 % _timestamp‚1.7844916122649043e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 _step‚793 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9413798487040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚605839483217920 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675e‹,”î'²èO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76417550 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99107899 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚759877369682432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚750463230409216 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52032630 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚596136571630080 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚605839329817600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9413798487040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚9799 % _timestamp‚1.7844916122649043e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 _step‚793 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9413798487040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚605839483217920 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9702887163392 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚750463212890624 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9414139273216 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76417550 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52032632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25940505 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚596136596054528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚750463230409216 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99097756 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚759877011377664 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99107899 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚751995198707844 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175525451 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚759877011377664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9414139273216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚742598325290640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9397213340526 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322™î ¶2³- _wandbruntime‚5089  _runtime‚5089þ[¸k2´ìM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9703855171584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚752069909976354 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9398141563874 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99107631 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚742671768412480 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 % _timestamp‚1.7844916126378798e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚750537422178304 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚750537404659712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52037713 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76425052 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚750537404659712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141551616 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175542842 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76425052 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚605899161582592 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚759952491369984 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9415069191680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚742671751373120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚596195472387072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77980762 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚752069553013560 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76425054 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25943352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚759952491369984 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99117788 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175542842 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175532683 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9415069191680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 165976064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 [GDeviceStatsMonitor.on_train_b›A¿ZÀ atch_end/active_bytes.large_pool.allocated‚750537422178304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99117788 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52037711 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25943051 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9414728398336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚605899327558656 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 trainer/global_step‚9799 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚596195447962624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚301 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99107631 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76425054 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 5089.21306318 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 _step‚794 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9703713619968ió +÷&µñM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚750537404659712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141551616 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175542842 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76425052 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚605899161582592 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚759952491369984 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9415069191680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚742671751373120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚596195472387072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77980762 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚752069553013560 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76425054 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25943352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚759952491369984 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99117788 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175542842 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175532683 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9415069191680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 165976064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚750537422178304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99117788 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52037711 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25943051 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9414728398336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚605899327558656 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 trainer/global_step‚9799 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚596195447962624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚301 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99107631 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76425054 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 5089.21306318 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 _step‚794 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9703713619968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9703855171584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚752069909976354 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9398141563874 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99107631 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚742671768412480 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 % _timestamp‚1.7844916126378798e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚750537422178304 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚750537404659712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52037713 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76425052 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720šrÈ2¶- _wandbruntime‚5089  _runtime‚5089…+¾4 ·„ _step‚795 1loss/SRqqt1/detection_loss‚0.7035573668438635 1loss/SRqqt2/detection_loss‚1.1627519558767663 2loss/SRqqt1/assignment_loss‚0.6613063426358589 2loss/SRqqt2/assignment_loss‚0.5479302622608423 3loss/FBt1/assignment_loss‚0.0001305612219398051 2loss/FBt2/assignment_loss‚0.004161860745697173 $ _timestamp‚1.784491612639664e+09 0loss/FRt1/assignment_loss‚0.7947621103535351 /loss/FRt1/detection_loss‚1.5171486360984099 /loss/FRt2/detection_loss‚0.5247794842845737 /loss/FBt1/detection_loss‚0.3475939614540299 %loss/total_loss‚8.598506501299159 trainer/global_step‚9799 0loss/FRt2/assignment_loss‚1.7786668648448172 /loss/FBt2/detection_loss‚0.5557170946788239 epoch‚12 _runtime‚5089.214472912(^µ¸‰ 0loss/FRt2/assignment_loss‚1.7786668648448172 /loss/FBt2/detection_loss‚0.5557170946788239 epoch‚12 _runtime‚5089.214472912 _step‚795 1loss/SRqqt1/detection_loss‚0.7035573668438635 1loss/SRqqt2/detection_loss‚1.1627519558767663 2loss/SRqqt1/assignment_loss‚0.6613063426358589 2loss/SRqqt2/assignment_loss‚0.5479302622608423 3loss/FBt1/assignment_loss‚0.0001305612219398051 2loss/FBt2/assignment_loss‚0.004161860745697173 $ _timestamp‚1.784491612639664e+09 0loss/FRt1/assignment_loss‚0.7947621103535351 /loss/FRt1/detection_loss‚1.5171486360984099 /loss/FRt2/detection_loss‚0.5247794842845737 /loss/FBt1/detection_loss‚0.3475939614540299 %loss/total_loss‚8.598506501299159 trainer/global_step‚9799›#_¯i¹‚(:Þ çÜôÒݸÙÌ cpu‚26.135534491075113% disk./.usageGB‚1173.7566452026367 disk.md0.in‚0 disk.md0.out‚ 133.1796875 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375$ memory_percent‚95.50633430480957# proc.memory.rssMB‚ 2840.23046875 network.recv‚ 1.375131e+06 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚14940.21923828125 disk.nvme1n1.in‚0& disk.nvme1n1.out‚14940.21923828125 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚52434.27197265625 disk.dm-1.in‚0 network.sent‚ 1.3446665e+07 disk.dm-1.out‚ 8112.62109375 disk.rbd0.in‚0' disk.nvme0c0n1.out‚45893.0654296875% disk.nvme0n1.out‚45893.0654296875' proc.memory.availableMB‚ 736.2421875 proc.cpu.threads‚17( disk./.usagePercent‚76.4537792022561 disk.rbd0.out‚ 0.4697265625( proc.memory.percent‚17.3353910446167¨Ñ䞺‚(:“ çÜôÒ°ÍçË gpu.0.gpu‚98.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚267.053' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.01766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†‘|2»- _wandbruntime‚5113  _runtime‚5113[åȼ ' lr-AdamW/pg1‚0.00018613803205506552 ' lr-AdamW/pg2‚0.00018613803205506552 trainer/global_step‚9849 $ _timestamp‚1.784491637375558e+09 _runtime‚5113.950559741 _step‚796Í^ÚºÍ½Ç ' lr-AdamW/pg2‚0.00018613803205506552 trainer/global_step‚9849 $ _timestamp‚1.784491637375558e+09 _runtime‚5113.950559741 _step‚796 ' lr-AdamW/pg1‚0.00018613803205506552œ‹:£2¾- _wandbruntime‚5113  _runtime‚5113ulé'¿ãO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9460667095040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 _runtime‚5113.951558803 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76792652 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚763633485958656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52286680 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76792650 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78354084 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124781568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 176386008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52286682 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚754172818863616 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76792650 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 trainer/global_step‚9849 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9460326308864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9460326308864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99603501 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚755712945896852 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚763633485958656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26067404 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚746269646454000 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9443316482212 _step‚797 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76792652 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚754172818863616 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99593358 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚608831093447168 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99593358 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚763633127653888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 176396153 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206016 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚754172801345024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78354368 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446!?`^ºÀèO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚763633485958656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26067404 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚746269646454000 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9443316482212 _step‚797 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76792652 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚754172818863616 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99593358 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚608831093447168 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99593358 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚763633127653888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 176396153 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206016 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚754172801345024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78354368 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9460667095040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 _runtime‚5113.951558803 PCDeviceStatsMonito1Cž?4r.on_train_batch_start/active.large_pool.allocated‚76792652 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚763633485958656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52286680 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76792650 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78354084 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124781568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 176386008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52286682 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚754172818863616 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76792650 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 trainer/global_step‚9849 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9460326308864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9460326308864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99603501 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚755712945896852 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351> J2Á- _wandbruntime‚5114  _runtime‚5114í9ó&ÂíM YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚754246993114112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚763708249334272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176413544 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 176403385 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚754246993114112 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 176413544 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99613390 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚599139264589824 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9751661355520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78361984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9461256220160 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78361697 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚754247010632704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9461597013504 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚755787300202568 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 _runtime‚5114.323946558 _step‚798 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52291761 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 176403385 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99613390 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76800154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99603233 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚763708607646208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚599139289014272 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26069936 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚746343055497120 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76800152 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚755787657165362 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9444584628882 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚763708607646208 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚608890925945344 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76800154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9461597013504 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚763708607646208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚599139289014272 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26069936 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚746343055497120 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76800152 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚755787657165362 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9444584628882 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚763708607646208 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚608890925945344 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76800154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9461597013504 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176413544 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 176403385 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚754246993114112 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 176413544 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99613390 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚599139264589824 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9751661355520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78361984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9461256220160 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78361697 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚754247010632704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9461597013504 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚755787300202568 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 _runtime‚5114.323946558 _step‚798 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52291761 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 176403385 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99613390 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76800154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99603233 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PÄ[mÊ‚(:â …ÝôÒÈ“çË network.recv‚ 1.383458e+06$ disk./.usageGB‚1174.008213043213 disk.md0.out‚ 134.05078125 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0 proc.cpu.threads‚17 network.sent‚ 1.3573439e+07) disk./.usagePercent‚76.47016531790997 disk.dm-1.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚46074.65478515625* proc.memory.percent‚17.337679862976074 disk.dm-0.in‚ 4.3359375" disk.dm-0.out‚52641.6357421875 disk.rbd0.out‚ 0.4697265625 disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚46074.65478515625' disk.nvme1c1n1.out‚14968.3681640625% disk.nvme1n1.out‚14968.3681640625$ memory_percent‚95.50886154174805 disk.md0.in‚0" disk.dm-1.out‚8113.13818359375 disk.nvme1n1.in‚0& proc.memory.availableMB‚ 735.828125# proc.memory.rssMB‚ 2840.60546875 cpu‚26.133392426111445¨!~`2Ë- _wandbruntime‚5139  _runtime‚5139Âp0«ÆÌÀ & lr-AdamW/pg1‚0.0001858551952359555 & lr-AdamW/pg2‚0.0001858551952359555 trainer/global_step‚9899 $ _timestamp‚1.784491662463605e+09 _runtime‚5139.038573921 _step‚800‹xå¦ËÍÅ & lr-AdamW/pg1‚0.0001858551952359555 & lr-AdamW/pg2‚0.0001858551952359555 trainer/global_step‚9899 $ _timestamp‚1.784491662463605e+09 _runtime‚5139.038573921 _step‚800 4¸Õ¿2Î- _wandbruntime‚5139  _runtime‚5139Ñ?â“ìÏçO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚611822900146688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚767389602234880 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚767389602234880 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9506854130688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26194080 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚602024204884480 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9798822140928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9798695262208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚757882407318016 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100088960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚759431050048542 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚9899 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚757882389799424 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9489759547220 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9507194916864 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚767389243930112 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77167752 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9507194916864 _step‚801 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚602024229308928 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177256710 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚767389243930112 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77167750 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 % _timestamp‚1.7844916624645886e+09 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚5139.039527692 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100099103 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78735100 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77167750 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78734810 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100088960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9506854130688 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚757882389799424 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚757882407318016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52540730àƒð÷ò'ÐìO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100099103 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77167752 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚5139.039527692 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100099103 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78735100 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77167750 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78734810 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100088960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9506854130688 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚757882389799424 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚757882407318016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52540730 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚611822900146688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚767389602234880 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚767389602234880 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9506854130688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26194080 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚602024204884480 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9798822140928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9798695262208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚757882407318016 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100088960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚759431050048542 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚9899 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚757882389799424 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9489759547220 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9507194916864 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚767389243930112 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77167752 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9507194916864 _step‚801 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚602024229308928 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177256710 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚767389243930112 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77167750 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 % _timestamp‚1.7844916624645886e+09 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 147101696 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77175252 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚750014359621120 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚757956599087104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚757956581568512 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9799657696768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 trainer/global_step‚9899 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77175252 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100098835 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚759505404354370 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚757956599087104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26196882 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9491027693890 % _timestamp‚1.7844916628361568e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177274087 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚767464365610496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77175254 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚602083081217024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177284246 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77175254 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100098835 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78742695 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚802 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚767464723922432 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100108992 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9507784041984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177284246 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚602083105641472 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚750014376660480 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 177274087 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26196603 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9490687770456 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚611882738913792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9799780374016 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚611882886015488 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚759505047391576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100108992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52545811 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52545813 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 _runtime‚ 5139.41136472 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚757956581568512 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78742414 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122677248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0Ý^…ÂÓõM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77175254 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100098835 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78742695 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚802 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚767464723922432 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100108992 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9507784041984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177284246 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚602083105641472 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚750014376660480 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 177274087 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26196603 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9490687770456 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚611882738913792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9799780374016 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚611882886015488 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚759505047391576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100108992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52545811 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52545813 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 _runtime‚ 5139.41136472 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚757956581568512 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78742414 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122677248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚1ä©á97518592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 147101696 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77175252 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚750014359621120 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚757956599087104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚757956581568512 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9799657696768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 trainer/global_step‚9899 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77175252 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100098835 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚759505404354370 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚757956599087104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26196882 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9491027693890 % _timestamp‚1.7844916628361568e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177274087 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚767464365610496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77175254 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚602083081217024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177284246 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100584562 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52794782 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9846813035008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚763148440274868 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100584562 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77542850 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9553381952512 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26319773 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9846684059136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9553722738688 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9536202612228 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚771145360206336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9553381952512 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚753612237662640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100594705 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚761591995772416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚761591978253824 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178137557 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚761591995772416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚771145718511104 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52794780 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100594705 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79114834 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 _runtime‚5163.840780376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9553722738688 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚771145718511104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚614814705570816 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77542852 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚614814858971136 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100584562 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52794782 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9846813035008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚763148440274868 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100584562 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77542850 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9553381952512 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26319773 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9846684059136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9553722738688 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9536202612228 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚771145360206336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9553381952512 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚753612237662640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100594705 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚761591995772416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚761591978253824 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178137557 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚761591995772416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚771145718511104 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52794780 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100594705 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79114834 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 _runtime‚5163.840780376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9553722738688 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚771145718511104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚614814705570816 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77542852 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚614814858971136 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844916879532375e+09 _step‚806 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77550352 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚605026897844224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77550354 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52799863 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77550352 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 _runtime‚5164.528418055 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚761666187541504 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚614874694431744 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100594437 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 984£gôø7641097216 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9537130835464 trainer/global_step‚9949 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚614874538941440 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚763222794580584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9537470758898 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100604594 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚771220840198656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚761666170022912 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚605026922268672 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚753685680784480 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚771220840198656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚761666170022912 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9554311863808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79122096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77550354 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚763223151543378 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26322520 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178144789 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178154948 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚771220481886720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9554311863808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52799861 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520#÷Øü&ãöM TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚771220840198656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚761666170022912 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9554311863808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79122096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77550354 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚763223151543378 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26322520 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178144789 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178154948 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚771220481886720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9554311863808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52799861 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9847772163072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7844916879532375e+09 _step‚806 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77550352 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚605026897844224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77550354 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52799863 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77550352 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 _runtime‚5164.528418055 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚761666187541504 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚614874694431744 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100594437 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9847641097216 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9537130835464 trainer/global_step‚9949 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚614874538941440 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚763222794580584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9537470758898 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100604594 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚771220840198656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚761666170022912 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚605026922268672 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚753685680784480 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968¦†:I 2ä- _wandbruntime‚5164  _runtime‚5164È»°H å… 2loss/SRqqt1/assignment_loss‚0.5270671670929433 3loss/FBt1/assignment_loss‚0.0003846371388018916 epoch‚12 % _timestamp‚1.7844916879543164e+09 /loss/FBt2/detection_loss‚0.8207829569000751 trainer/global_step‚9949 _runtime‚5164.529028386 3loss/FBt2/assignment_loss‚0.0018987936744258502 /loss/FRt1/detection_loss‚1.4391074005591993 /loss/FRt2/detection_loss‚0.5178005710397178 0loss/SRqqt1/detection_loss‚0.615434194032637 0loss/SRqqt2/detection_loss‚0.877453704348866 2loss/SRqqt2/assignment_loss‚0.6336445623955287 0loss/FBt1/detection_loss‚0.34716387323242726 %loss/total_loss‚8.694469414764256 _step‚807 0loss/FRt1/assignment_loss‚0.9176206437475753 0loss/FRt2/assignment_loss‚1.9961109106020587ëÂäXæŠ /loss/FBt2/detection_loss‚0.8207829569000751 trainer/global_step‚9949 _runtime‚5164.529028386 3loss/FBt2/assignment_loss‚0.0018987936744258502 /loss/FRt1/detection_loss‚1.4391074005591993 /loss/FRt2/detection_loss‚0.5178005710397178 0loss/SRqqt1/detection_loss‚0.615434194032637 0loss/SRqqt2/detection_loss‚0.877453704348866 2loss/SRqqt2/assignment_loss‚0.6336445623955287 0loss/FBt1/detection_loss‚0.34716387323242726 %loss/total_loss‚8.694469414764256 _step‚807 0loss/FRt1/assignment_loss‚0.9176206437475753 0loss/FRt2/assignment_loss‚1.9961109106020587 2loss/SRqqt1/assignment_loss‚0.5270671670929433 3loss/FBt1/assignment_loss‚0.0003846371388018916 epoch‚12 % _timestamp‚1.7844916879543164e+09§‹P¦jç‚(:ß ²ÝôÒºþ–Ì disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1n1.in‚0 disk.md0.in‚0# proc.memory.rssMB‚ 2840.73046875 cpu‚26.130304432296906 network.sent‚ 1.3655567e+07 network.recv‚ 1.393724e+06% disk./.usageGB‚1172.0115394592285 disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚46144.14404296875 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚15704.87939453125 disk.dm-0.in‚ 4.3359375( disk.nvme0c0n1.out‚46144.14404296875 disk.md0.out‚ 134.96484375" disk.dm-0.out‚53445.0029296875 disk.dm-1.in‚0 disk.dm-1.out‚ 8113.78515625& proc.memory.availableMB‚ 735.734375( proc.memory.percent‚17.3384428024292( disk./.usagePercent‚76.3401100445679 disk.rbd0.in‚0( disk.nvme1c1n1.out‚15704.87939453125$ memory_percent‚95.50943374633789 proc.cpu.threads‚17Õ~ œè‚(:‘ ²ÝôÒ‚ÈÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚52.0 gpu.0.powerWatts‚118.589' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.52966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úžÉà2é- _wandbruntime‚5188  _runtime‚5188H;Éêà _step‚808 ' lr-AdamW/pg1‚0.00018528570942311288 ' lr-AdamW/pg2‚0.00018528570942311288 trainer/global_step‚9999 % _timestamp‚1.7844917122634325e+09 _runtime‚5188.838406601¡|(ÎëÈ _step‚808 ' lr-AdamW/pg1‚0.00018528570942311288 ' lr-AdamW/pg2‚0.00018528570942311288 trainer/global_step‚9999 % _timestamp‚1.7844917122634325e+09 _runtime‚5188.838406601¨fb?2ì- _wandbruntime‚5188  _runtime‚5188ý¥ï9í'íçO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178998114 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚774901476482560 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26445419 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 _runtime‚5188.839282642 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77917950 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26445693 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚607911862563328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚774901834787328 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚765301566708224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101080164 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9582985600558 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53048832 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9600250560512 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚757283541786640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚765301584226816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79494249 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 % _timestamp‚1.7844917122644157e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚774901476482560 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77917952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚617806628024832 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚774901834787328 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚765301566708224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101080164 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9582985600558 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53048832 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9600250560512 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚757283541786640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚765301584226816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79494249 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 % _timestamp‚1.7844917122644157e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚774901476482560 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77917952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚617806628024832 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚9999 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚765375758477312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 179025650 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9895591776768 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚617866463835648 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9600839685632 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 179015491 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77925452 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9583913823906 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 179015491 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚766940541769592 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 _step‚810 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101100196 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚774976956474880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77925454 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26448204 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 179025650 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101100196 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚757356984908480 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53053911 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚766940898732386 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26447923 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133163008 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9583573900472 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9600839685632 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79501834 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77925452 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9601180478976 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚757356967869120 _runtime‚5189.210046176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚774976956474880 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚765375775995904”i²ü&ñöM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚774976956474880 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚765375775995904 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚9999 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚765375758477312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 179025650 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9895591776768 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚617866463835648 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9600839685632 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 179015491 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77925452 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9583913823906 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 179015491 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚766940541769592 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 _step‚810 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101100196 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚774976956474880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77925454 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26448204 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 179025650 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101100196 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚757356984908480 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53053911 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚766940898732386 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26447923 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133163008 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9583573900472 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9600839685632 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79501834 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77925452 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9601180478976 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚757356967869120 _runtime‚5189.210046176ª=Sh2ò- _wandbruntime‚5189  _runtime‚51896é ó„ 2loss/FBt2/assignment_loss‚0.002280672818396156 /loss/FRt1/detection_loss‚1.2574868456970525 1loss/SRqqt1/detection_loss‚0.5524153546540118 1loss/SRqqt2/detection_loss‚0.8618229282337722 0loss/FRt2/assignment_loss‚1.8220910425927188 2loss/SRqqt2/assignment_loss‚0.6448599517817184 /loss/FRt2/detection_loss‚0.5291781553569636 /loss/FBt1/detection_loss‚0.5006233219161857 %loss/total_loss‚8.029017334694561 _step‚811 2loss/FBt1/assignment_loss‚0.011866328604009088 trainer/global_step‚9999 _runtime‚5189.210685077 0loss/FRt1/assignment_loss‚0.6803302853981371 2loss/SRqqt1/assignment_loss‚0.6333962283412499 /loss/FBt2/detection_loss‚0.5326662193003454 epoch‚12 % _timestamp‚1.7844917126359823e+09‡eÂô‰ 0loss/FRt1/assignment_loss‚0.6803302853981371 2loss/SRqqt1/assignment_loss‚0.6333962283412499 /loss/FBt2/detection_loss‚0.5326662193003454 epoch‚12 % _timestamp‚1.7844917126359823e+09 2loss/FBt2/assignment_loss‚0.002280672818396156 /loss/FRt1/detection_loss‚1.2574868456970525 1loss/SRqqt1/detection_loss‚0.5524153546540118 1loss/SRqqt2/detection_loss‚0.8618229282337722 0loss/FRt2/assignment_loss‚1.8220910425927188 2loss/SRqqt2/assignment_loss‚0.6448599517817184 /loss/FRt2/detection_loss‚0.5291781553569636 /loss/FBt1/detection_loss‚0.5006233219161857 %loss/total_loss‚8.029017334694561 _step‚811 2loss/FBt1/assignment_loss‚0.011866328604009088 trainer/global_step‚9999 _runtime‚5189.210685077«†Óÿ;hõ‚(:Ý ÁÝôÒ°ÄþË network.recv‚ 1.397881e+06$ disk.nvme0n1.out‚46151.810546875 disk.nvme1c1n1.in‚0# disk.dm-0.out‚53454.06689453125( proc.memory.availableMB‚ 488.95703125# proc.memory.rssMB‚ 2840.73046875 disk.rbd0.in‚0& disk.nvme0c0n1.out‚46151.810546875 disk.nvme0n1.in‚ 4.3359375% disk.nvme1n1.out‚15707.0185546875$ memory_percent‚97.01564311981201 cpu‚26.129114158126267' disk.nvme1c1n1.out‚15707.0185546875 network.sent‚ 1.371792e+07) disk./.usagePercent‚76.34010706287975% disk./.usageGB‚1172.0114936828613 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375 disk.md0.in‚0" disk.dm-1.out‚8113.85302734375 disk.nvme1n1.in‚0 disk.md0.out‚ 135.265625 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0( proc.memory.percent‚17.3384428024292 proc.cpu.threads‚17³\rœö‚(:‘ ÁÝôÒ¸›Î gpu.0.gpu‚7.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚52.0 gpu.0.powerWatts‚118.264' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.42133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éÕ-p÷‚(:å ÐÝôÒß¹ôË proc.cpu.threads‚17 network.sent‚ 1.3767398e+07% disk./.usageGB‚1172.0115356445312& disk.nvme1n1.out‚15707.77880859375 disk.dm-1.in‚0 network.recv‚ 1.402102e+06 disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚46153.29150390625 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚15707.77880859375 disk.nvme1n1.in‚0# disk.dm-0.out‚53455.11181640625 disk.rbd0.out‚ 0.4697265625) disk./.usagePercent‚76.34010979609388 disk.nvme0c0n1.in‚ 4.3359375 disk.md0.in‚0 disk.md0.out‚ 135.7734375" disk.dm-1.out‚8113.96630859375 disk.rbd0.in‚0$ memory_percent‚97.01292514801025( proc.memory.availableMB‚ 489.40234375( disk.nvme0c0n1.out‚46153.29150390625 disk.dm-0.in‚ 4.3359375# proc.memory.rssMB‚ 2840.73046875( proc.memory.percent‚17.3384428024292 cpu‚26.128169060734415ômdø‚(:’ ÐÝôÒ“±¸Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚118.396' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.465333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï"2ù- _wandbruntime‚5213  _runtime‚5213@s¹hÊúÄ ' lr-AdamW/pg1‚0.00018499906955085141 ' lr-AdamW/pg2‚0.00018499906955085141 trainer/global_step‚10049 % _timestamp‚1.7844917372449274e+09 _runtime‚5213.819918377 _step‚812\A(ÏûÉ _step‚812 ' lr-AdamW/pg1‚0.00018499906955085141 ' lr-AdamW/pg2‚0.00018499906955085141 trainer/global_step‚10049 % _timestamp‚1.7844917372449274e+09 _runtime‚5213.819918377¬S»©R2ü- _wandbruntime‚5213  _runtime‚5213Ù°-å–ýçO WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocateÏ€œßW&d‚ 179878961 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚760954845910640 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9629088742244 _runtime‚5213.820824928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9646778382336 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚610855654766080 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 _step‚813 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚778657592758784 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9646778382336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26572690 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101575766 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚760954862950000 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101575766 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159691776 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9942585081344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 179868816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101585909 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚769011172681216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚610855679190528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78293052 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚769011155162624 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53302882 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78293050 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9942720348672 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚778657592758784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚770583934652884 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚610855654766080 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 _step‚813 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚778657592758784 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9646778382336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26572690 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101575766 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚760954862950000 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101575766 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159691776 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9942585081344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 179868816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101585909 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚769011172681216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚610855679190528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78293052 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚769011155162624 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53302882 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78293050 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9942720348672 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚778657592758784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚770583934652884 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 179878961 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968­ÿg2ÿ- _wandbruntime‚5214  _runtime‚5214>Õø&€òM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚610914555523072 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚769085346931712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9630016965480 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚778733072751104 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚778733072751104 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 179886193 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 % _timestamp‚1.7844917376156414e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79883404 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚769085346931712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚761028289032480 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133163008 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚770658645921394 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78300552 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101585641 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9647367507456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26575441 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9630356888914 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5214.190805985 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78300554 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚610914531098624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 179896352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚620858074979328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 179896352 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101595798 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚769085364450304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 179886193 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚770658288958600 _step‚814 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79883121 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968wJ‰Nj ÷M WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚778733072751104 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 179886193 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 % _timestamp‚1.7844917376156414e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79883404 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚769085346931712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚761028289032480 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133163008 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚770658645921394 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78300552 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101585641 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9647367507456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26575441 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9630356888914 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5214.190805985 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78300554 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚610914531098624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 179896352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚620858074979328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 179896352 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101595798 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚769085364450304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 179886193 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚770658288958600 _step‚814 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79883121 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚610914555523072 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚769085346931712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9630016965480 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚778733072751104 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344®¸þ0Y2‚- _wandbruntime‚5214  _runtime‚5214Qä¼ ƒ… 3loss/FBt1/assignment_loss‚0.0015390795530909188 epoch‚12 % _timestamp‚1.7844917376167014e+09 _step‚815 2loss/SRqqt1/assignment_loss‚0.4725935171070002 /loss/FRt2/detection_loss‚0.5164387160533369 1loss/SRqqt2/detection_loss‚0.8677708170582334 0loss/FBt1/detection_loss‚0.44149969740472034 1loss/FBt2/assignment_loss‚0.03041759081027558 /loss/FRt1/detection_loss‚1.3662703676751429 1loss/SRqqt1/detection_loss‚0.5985603058685524 $loss/total_loss‚8.34333810247935 0loss/FRt1/assignment_loss‚0.6906495659226873 2loss/SRqqt2/assignment_loss‚0.6253099952608001 /loss/FBt2/detection_loss‚0.8695080870141586 trainer/global_step‚10049 _runtime‚5214.194397441 0loss/FRt2/assignment_loss‚1.8627803627513508Cl(„Š 2loss/SRqqt1/assignment_loss‚0.4725935171070002 /loss/FRt2/detection_loss‚0.5164387160533369 1loss/SRqqt2/detection_loss‚0.8677708170582334 0loss/FBt1/detection_loss‚0.44149969740472034 1loss/FBt2/assignment_loss‚0.03041759081027558 /loss/FRt1/detection_loss‚1.3662703676751429 1loss/SRqqt1/detection_loss‚0.5985603058685524 $loss/total_loss‚8.34333810247935 0loss/FRt1/assignment_loss‚0.6906495659226873 2loss/SRqqt2/assignment_loss‚0.6253099952608001 /loss/FBt2/detection_loss‚0.8695080870141586 trainer/global_step‚10049 _runtime‚5214.194397441 0loss/FRt2/assignment_loss‚1.8627803627513508 3loss/FBt1/assignment_loss‚0.0015390795530909188 epoch‚12 % _timestamp‚1.7844917376167014e+09 _step‚815¯n¨É Ÿ…‚(:” ßÝôÒ˱¹Ì gpu.0.gpu‚96.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚168.134' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚56.044666666666664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž8õMj†‚(:ß ßÝôÒØæÍ disk.rbd0.in‚0& disk.nvme0c0n1.out‚46424.896484375 disk.nvme0n1.in‚ 4.3359375( disk.nvme1c1n1.out‚15708.50732421875 disk.dm-0.in‚ 4.3359375$ memory_percent‚97.01483249664307( proc.memory.percent‚17.3384428024292 cpu‚26.12720875297745% disk./.usageGB‚1172.2740592956543 disk.nvme0c0n1.in‚ 4.3359375$ disk.nvme0n1.out‚46424.896484375 disk.md0.out‚ 136.11328125" disk.dm-0.out‚53726.5654296875( proc.memory.availableMB‚ 489.08984375# proc.memory.rssMB‚ 2840.73046875 proc.cpu.threads‚17 network.recv‚ 1.405306e+06) disk./.usagePercent‚76.35720952910955 disk.rbd0.out‚ 0.4697265625 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚15708.50732421875 network.sent‚ 1.377341e+07 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-1.in‚0" disk.dm-1.out‚8114.09814453125õ Ëj‡‚(:ß îÝôÒÁÇíË disk.md0.in‚0 disk.dm-0.in‚ 4.3359375" disk.dm-1.out‚8114.17236328125& proc.memory.availableMB‚ 489.203125& disk.nvme0n1.out‚46425.78662109375 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚15709.0302734375 disk.dm-1.in‚0$ memory_percent‚97.01414108276367( proc.memory.percent‚17.3384428024292 cpu‚26.126151102102657 network.sent‚ 1.3836648e+07) disk./.usagePercent‚76.43982664107345 disk.rbd0.in‚0( disk.nvme0c0n1.out‚46425.78662109375 disk.nvme0n1.in‚ 4.3359375 disk.nvme1n1.in‚0 disk.md0.out‚ 136.46484375" disk.dm-0.out‚53727.1474609375 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375' disk.nvme1c1n1.out‚15709.0302734375# proc.memory.rssMB‚ 2840.73046875 proc.cpu.threads‚17 network.recv‚ 1.409956e+06# disk./.usageGB‚1173.54243850708\³Tôžˆ‚(:“ îÝôÒ°ä£Ì gpu.0.gpu‚59.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚251.623' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚83.87433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íö<¾2‰- _wandbruntime‚5238  _runtime‚5238 ä£ÉŠÃ _step‚816 ' lr-AdamW/pg1‚0.00018471117114944518 ' lr-AdamW/pg2‚0.00018471117114944518 trainer/global_step‚10099 $ _timestamp‚1.784491762139055e+09 _runtime‚5238.714021157|yéSÎ‹È ' lr-AdamW/pg1‚0.00018471117114944518 ' lr-AdamW/pg2‚0.00018471117114944518 trainer/global_step‚10099 $ _timestamp‚1.784491762139055e+09 _runtime‚5238.714021157 _step‚816°qo—a2Œ- _wandbruntime‚5238  _runtime‚5238³ÝsSî'èO trainer/global_step‚10099 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚772720743617024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78668152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚774302038804574 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚613799471393280 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78668150 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102081511 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚772720761135616 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 _step‚817 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚296 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180749663 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9675531807252 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚782413709035008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53556932 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9990516174848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚782413709035008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚613799495817728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚764626167074000 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78668152 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102071368 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚772720743617024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9692965417984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7844917621400747e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚764626150034640 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180739518 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚774301681841892 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9693306204160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102071368 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80256332 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9990645150720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9693306204160 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚772720761135616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9692965417984 _runtime‚5238.715014079 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675¹¿~¦ó'ŽíO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9693306204160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102071368 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80256332 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9990645150720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9693306204160 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚772720761135616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9692965417984 _runtime‚5238.715014079 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 trainer/global_step‚10099 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚772720743617024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78668152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚774302038804574 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚613799471393280 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78668150 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102081511 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚772720761135616 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 _step‚817 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚296 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180749663 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9675531807252 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚782413709035008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53556932 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9990516174848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚782413709035008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚613799495817728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚764626167074000 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78668152 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102071368 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚772720743617024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9692965417984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7844917621400747e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚764626150034640 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180739518 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚774301681841892 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260±Õn92- _wandbruntime‚5239  _runtime‚5239¾P‘ñM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78675654 $ _timestamp‚1.784491762508997e+09 E?DeviceStatsMo “f&nitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102091400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102091400 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9694236122624 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚772794952904704 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53562013 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚774376036147608 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚772794952904704 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚613858372150272 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9991600252416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚772794935386112 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9991477575168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80264250 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 147101696 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9676460030488 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚782489189027328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9693895329280 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180767054 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102081243 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180756895 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚764699576117120 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 trainer/global_step‚10099 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 _step‚818 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80263967 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78675652 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53562011 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚774376393110402 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26702237 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚613858347725824 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5239.084139055 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚772794935386112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚623849825300992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122677248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9694236122624 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78675652 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚764699593156480 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180756895 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚623849972402688 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9676799953922 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9693895329280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260ä$™ü&‘öM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚623849825300992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 122677248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9694236122624 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78675652 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚764699593156480 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180756895 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚623849972402688 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9676799953922 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9693895329280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78675654 $ _timestamp‚1.784491762508997e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102091400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102091400 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9694236122624 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚772794952904704 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53562013 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚774376036147608 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚772794952904704 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚613858372150272 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9991600252416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚772794935386112 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9991477575168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80264250 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 147101696 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9676460030488 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚782489189027328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9693895329280 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180767054 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102081243 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180756895 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚764699576117120 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 trainer/global_step‚10099 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 _step‚818 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80263967 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78675652 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53562011 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚774376393110402 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26702237 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚613858347725824 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5239.084139055 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚772794935386112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935²*;ß2’- _wandbruntime‚5239  _runtime‚5239ñ…ß³ “… 0loss/FRt1/assignment_loss‚0.8013653682308947 /loss/FRt2/assignment_loss‚1.783996036660816 2loss/FBt1/assignment_loss‚0.012698794926401892 2loss/FBt2/assignment_loss‚0.023281973879181805 /loss/FRt1/detection_loss‚1.3923225256016551 0loss/FRt2/detection_loss‚0.47127939378650063 % _timestamp‚1.7844917625100322e+09 1loss/SRqqt1/detection_loss‚0.6740656116216658 1loss/SRqqt2/detection_loss‚0.7765959095040971 /loss/FBt1/detection_loss‚0.5518784150624505 /loss/FBt2/detection_loss‚0.9970606861636042 %loss/total_loss‚8.553831715116669 trainer/global_step‚10099 2loss/SRqqt1/assignment_loss‚0.4932554999656718 epoch‚12 _step‚819 2loss/SRqqt2/assignment_loss‚0.5760314997137307 _runtime‚5239.084692376~¤¹™”Š 2loss/SRqqt1/assignment_loss‚0.4932554999656718 epoch‚12 _step‚819 2loss/SRqqt2/assignment_loss‚0.5760314997137307 _runtime‚5239.084692376 0loss/FRt1/assignment_loss‚0.8013653682308947 /loss/FRt2/assignment_loss‚1.783996036660816 2loss/FBt1/assignment_loss‚0.012698794926401892 2loss/FBt2/assignment_loss‚0.023281973879181805 /loss/FRt1/detection_loss‚1.3923225256016551 0loss/FRt2/detection_loss‚0.47127939378650063 % _timestamp‚1.7844917625100322e+09 1loss/SRqqt1/detection_loss‚0.6740656116216658 1loss/SRqqt2/detection_loss‚0.7765959095040971 /loss/FBt1/detection_loss‚0.5518784150624505 /loss/FBt2/detection_loss‚0.9970606861636042 %loss/total_loss‚8.553831715116669 trainer/global_step‚10099³Ãgqq•‚(:æ ýÝôÒÛ±òË network.sent‚ 1.3852413e+07 disk.nvme0n1.in‚ 4.3359375 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0 cpu‚26.125173802478376 network.recv‚ 1.413246e+06% disk./.usageGB‚1174.2877655029297 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375& disk.nvme1n1.out‚15729.18798828125" disk.dm-0.out‚55648.1669921875$ memory_percent‚97.01540470123291) disk./.usagePercent‚76.48837423891416( disk.nvme0c0n1.out‚48327.92041015625& disk.nvme0n1.out‚48327.92041015625 disk.nvme1c1n1.in‚0( proc.memory.availableMB‚ 488.99609375# proc.memory.rssMB‚ 2840.85546875* proc.memory.percent‚17.339205741882324( disk.nvme1c1n1.out‚15729.18798828125 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 136.99609375! disk.dm-1.out‚8114.3251953125 proc.cpu.threads‚17¢ðN ž–‚(:“ ýÝôÒæä§Ì gpu.0.gpu‚91.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚252.351' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.11699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö˜¢G2—- _wandbruntime‚5263  _runtime‚5263„ó1ɘà _step‚820 ' lr-AdamW/pg1‚0.00018442201883017145 ' lr-AdamW/pg2‚0.00018442201883017145 trainer/global_step‚10149 $ _timestamp‚1.784491787089247e+09 _runtime‚5263.664213849Ÿ×-±Î™È ' lr-AdamW/pg1‚0.00018442201883017145 ' lr-AdamW/pg2‚0.00018442201883017145 trainer/global_step‚10149 $ _timestamp‚1.784491787089247e+09 _runtime‚5263.664213849 _step‚820´tlî92š- _wandbruntime‚5263  _runtime‚5263¨§·—ó$›êO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚821 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9739834025984 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚778019429030900 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚768297454158640 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 _runtime‚5263.665305061 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26826815 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79043252 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79043250 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9739493239808 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102577113 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53810980 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚786170183616000 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80637795 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102566970 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26827113 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚786170183616000 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚786169825311232 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚776430349590016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚616743312444928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚786169825311232 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10038639207936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚626781804544000 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 181610220 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122684416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9721974872260 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80638095 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79043252 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚778019785993582 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10149 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147108864 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10038516523520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9739834025984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102566970 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚298 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9722314795582 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9739493239808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚äÎLý 181620365 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79043250 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚300 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 % _timestamp‚1.7844917870902963e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102577113 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2.+„õ'œïO _step‚821 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9739834025984 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚778019429030900 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚768297454158640 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 _runtime‚5263.665305061 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26826815 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79043252 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79043250 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9739493239808 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102577113 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53810980 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚786170183616000 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80637795 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102566970 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26827113 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚786170183616000 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚786169825311232 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚776430349590016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚616743312444928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚786169825311232 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10038639207936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚626781804544000 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 181610220 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122684416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9721974872260 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80638095 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79043252 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚778019785993582 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10149 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147108864 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10038516523520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9739834025984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102566970 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚298 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9722314795582 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9739493239808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 181620365 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79043250 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚300 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 % _timestamp‚1.7844917870902963e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102577113 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1µ¿ž¤\2- _wandbruntime‚5264  _runtime‚5264²ù{ú&žôM K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102576845 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53816061 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10039599939584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚5264.036693288 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79050752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9722903095496 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 trainer/global_step‚10149 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚776504541359104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚786245305303552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181637756 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚616802164353024 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚768370897280480 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53816063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26829634 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9723243018930 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚786244946991616 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80645409 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚776504541359104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181637756 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 181627597 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚776504523840512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚776504523840512 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102587002 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26829348 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚778094140299410 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80645697 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 _step‚822 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79050754 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79050754 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚616802188777472 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚626841639518208 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9740423151104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9740763944448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10039475165184 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 % _timestamp‚1.7844917874615726e+09 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80645697 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 _step‚822 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79050754 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79050754 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚616802188777472 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚626841639518208 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9740423151104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9740763944448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10039475165184 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 % _timestamp‚1.7844917874615726e+09 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53816063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26829634 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9723243018930 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚786244946991616 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80645409 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚776504541359104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181637756 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 181627597 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚776504523840512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159¶ûYP2 - _wandbruntime‚5264  _runtime‚5264º<]K¡‚ epoch‚12 0loss/FRt1/assignment_loss‚0.8180475183253019 /loss/FRt2/detection_loss‚0.5713234763634637 _runtime‚5264.037388879 /loss/FRt2/assignment_loss‚1.829511470961989 2loss/SRqqt2/assignment_loss‚0.6134218430865745 /loss/FRt1/detection_loss‚1.2211889972101968 /loss/FBt2/detection_loss‚0.4282834141887724 2loss/SRqqt1/assignment_loss‚0.5048085935195268 1loss/FBt1/assignment_loss‚0.05947140751882106 2loss/FBt2/assignment_loss‚0.000286155108741184 1loss/SRqqt1/detection_loss‚0.5640140569073459 0loss/SRqqt2/detection_loss‚0.845273374481813 /loss/FBt1/detection_loss‚0.6142945022167017 trainer/global_step‚10149 %loss/total_loss‚8.069924809889248 % _timestamp‚1.7844917874626505e+09 _step‚823Fœá„ ¢‡ epoch‚12 0loss/FRt1/assignment_loss‚0.8180475183253019 /loss/FRt2/detection_loss‚0.5713234763634637 _runtime‚5264.037388879 /loss/FRt2/assignment_loss‚1.829511470961989 2loss/SRqqt2/assignment_loss‚0.6134218430865745 /loss/FRt1/detection_loss‚1.2211889972101968 /loss/FBt2/detection_loss‚0.4282834141887724 2loss/SRqqt1/assignment_loss‚0.5048085935195268 1loss/FBt1/assignment_loss‚0.05947140751882106 2loss/FBt2/assignment_loss‚0.000286155108741184 1loss/SRqqt1/detection_loss‚0.5640140569073459 0loss/SRqqt2/detection_loss‚0.845273374481813 /loss/FBt1/detection_loss‚0.6142945022167017 trainer/global_step‚10149 %loss/total_loss‚8.069924809889248 % _timestamp‚1.7844917874626505e+09 _step‚823·ÎZ½I]£‚(:ˆ ŒÞôÒ¦ÙõË gpu.0.gpu‚90.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚p|­617.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚132.252' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚44.084 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†ñC m¤‚(:â ŒÞôÒ¦ï¿Ì disk.nvme0c0n1.in‚ 4.3359375 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375! disk.dm-1.out‚8114.4033203125& proc.memory.availableMB‚ 488.765625 cpu‚26.1241961929987 disk.nvme0n1.in‚ 4.3359375 disk.nvme1n1.in‚0 disk.md0.out‚ 137.32421875" disk.dm-0.out‚55679.2880859375 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625* proc.memory.percent‚17.339205741882324( disk.nvme0c0n1.out‚48359.33056640625& disk.nvme0n1.out‚48359.33056640625 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚15729.68798828125& disk.nvme1n1.out‚15729.68798828125 disk.dm-1.in‚0 network.sent‚ 1.3914739e+07) disk./.usagePercent‚76.50810208155228% disk./.usageGB‚1174.5906372070312$ memory_percent‚97.01681137084961# proc.memory.rssMB‚ 2840.85546875 proc.cpu.threads‚17 network.recv‚ 1.417509e+06#9Um¥‚(:â ›ÞôÒË–ÍË disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375& disk.nvme1n1.out‚15899.99755859375 disk.dm-1.in‚0* proc.memory.percent‚17.339205741882324 cpu‚26.123097983543424 network.sent‚ 1.3964274e+07 network.recv‚ 1.421798e+06% disk./.usageGB‚1174.4971656799316( disk.nvme0c0n1.out‚49047.28759765625& disk.nvme0n1.out‚49047.28759765625 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚15899.99755859375 disk.rbd0.out‚ 0.4697265625 disk.nvme0n1.in‚ 4.3359375( proc.memory.availableMB‚ 488.73046875# proc.memory.rssMB‚ 2840.85546875 proc.cpu.threads‚17' disk./.usagePercent‚76.502013722842 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 137.55859375 disk.dm-0.in‚ 4.3359375! disk.dm-0.out‚56536.798828125! disk.dm-1.out‚8114.6259765625# memory_percent‚97.0170259475708‘,Û0Ÿ¦‚(:” ›ÞôÒËÍ–Ì gpu.0.gpu‚80.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚125.951' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚41.983666666666664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ™ÿÝ2§- _wandbruntime‚5288  _runtime‚5288ÿõI+欀 ' lr-AdamW/pg2‚0.00018413161722439158 trainer/global_step‚10199 % _timestamp‚1.7844918118070467e+09 _runtime‚5288.382014614 _step‚824 ' lr-AdamW/pg1‚0.00018413161722439158þåH|Ï©É _runtime‚5288.382014614 _step‚824 ' lr-AdamW/pg1‚0.00018413161722439158 ' lr-AdamW/pg2‚0.00018413161722439158 trainer/global_step‚10199 % _timestamp‚1.7844918118070467e+09¸|m³£2ª- _wandbruntime‚5288  _runtime‚5288w"Xð'«êO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103062572 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚789925941587456 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚780139938044416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79418352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497472 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚780139920525824 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54065030 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103062572 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9786361847808 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79418350 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26953572 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚619687104647680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚771968775322000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9786361847808 _runtime‚5288.386440351 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79418352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚781737533182590 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚629773747707904 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81018604 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9786021061632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9768417937268 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚789926299892224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54065032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚781737176219908 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚619687129072128 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10086618635776 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚780139938044416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26953287 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103072715 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 % _timestamp‚1.7844918118080456e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79418350 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚629773592210432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103072715 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 182491067 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9786361847808 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79418350 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26953572 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚619687104647680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚771968775322000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9786361847808 _runtime‚5288.386440351 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79418352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚781737533182590 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚629773747707904 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81018604 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9786021061632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9768417937268 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚789926299892224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54065032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚781737176219908 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚619687129072128 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10086618635776 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚780139938044416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26953287 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103072715 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 % _timestamp‚1.7844918118080456e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79418350 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚629773592210432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103072715 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 182491067 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚789925941587456 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚780139938044416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79418352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497472 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚780139920525824 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54065030 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103062572 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240¹‡¬Ù¹2­- _wandbruntime‚5288  _runtime‚5288¹\ ”é#®ôM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚790001421579776 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26955772 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79425854 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 182498299 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 _runtime‚5288.752200829 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9769686083938 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚772042201404480 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9769346160504 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182508458 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 _step‚826 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 % _timestamp‚1.7844918121768668e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚790001421579776 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚790001063267840 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182508458 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103082604 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚781811530525624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54070111 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚629833428497408 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79425852 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9787291766272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103072447 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚780214112294912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26956054 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚781811887488418 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10087578583040 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81026167 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚619745980980224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚780214112294912 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103072447 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 trainer/global_step‚10199 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79425852 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9787291766272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚780214129813504 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79425854 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81025883 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚772042184365120 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9786950972928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103082604 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.alg"~Jl.allocated‚629833583987712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10087447517184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54070113 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚619746005404672 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚781811887488418 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10087578583040 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81026167 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚619745980980224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚780214112294912 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103072447 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 trainer/global_step‚10199 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79425852 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9787291766272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚780214129813504 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79425854 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81025883 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚772042184365120 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9786950972928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103082604 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚629833583987712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10087447517184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54070113 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚619746005404672 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79425854 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 182498299 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 _runtime‚5288.752200829 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9769686083938 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚772042201404480 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9769346160504 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182508458 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 _step‚826 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 % _timestamp‚1.7844918121768668e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚790001421579776 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚790001063267840 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182508458 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103082604 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚781811530525624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54070111 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚629833428497408 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79425852 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9787291766272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103072447 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚780214112294912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26956054 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448ºx„*_2°- _wandbruntime‚5288  _runtime‚5288!±a^ ±„ .loss/FRt2/assignment_loss‚1.79953338214882 2loss/FBt2/assignment_loss‚0.017094744589848786 .loss/FRt2/detection_loss‚0.563956803733421 /loss/FBt2/detection_loss‚1.2221155219711364 trainer/global_step‚10199 /loss/FRt1/detection_loss‚1.2736185579726347 0loss/FBt1/detection_loss‚0.44393547115249754 epoch‚12 % _timestamp‚1.7844918121781085e+09 _step‚827 2loss/SRqqt1/assignment_loss‚0.4718665598669769 2loss/SRqqt2/assignment_loss‚0.6955664546915795 3loss/FBt1/assignment_loss‚0.0019978537929576834 1loss/SRqqt1/detection_loss‚0.5870762555906682 _runtime‚5288.753008631 0loss/FRt1/assignment_loss‚0.8376577176124211 1loss/SRqqt2/detection_loss‚0.9139012960996479 %loss/total_loss‚8.828320619222609ҜϞ²‰ .loss/FRt2/assignment_loss‚1.79953338214882 2loss/FBt2/assignment_loss‚0.017094744589848786 .loss/FRt2/detection_loss‚0.563956803733421 /loss/FBt2/detection_loss‚1.2221155219711364 trainer/global_step‚10199 /loss/FRt1/detection_loss‚1.2736185579726347 0loss/FBt1/detection_loss‚0.44393547115249754 epoch‚12 % _timestamp‚1.7844918121781085e+09 _step‚827 2loss/SRqqt1/assignment_loss‚0.4718665598669769 2loss/SRqqt2/assignment_loss‚0.6955664546915795 3loss/FBt1/assignment_loss‚0.0019978537929576834 1loss/SRqqt1/detection_loss‚0.5870762555906682 _runtime‚5288.753008631 0loss/FRt1/assignment_loss‚0.8376577176124211 1loss/SRqqt2/detection_loss‚0.9139012960996479 %loss/total_loss‚8.828320619222609»~’¬«n³‚(:ã ªÞôÒ’úÿË network.recv‚ 1.425413e+06 disk.md0.in‚0 disk.dm-1.in‚0! disk.dm-1.out‚8114.7353515625 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0$ memory_percent‚97.01733589172363 disk.md0.out‚ 138.0859375 disk.dm-0.in‚ 4.3359375! disk.dm-0.out‚56537.630859375 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚49048.54736328125& disk.nvme1n1.out‚15900.80029296875) disk./.usagePercent‚76.50208950741558% disk./.usageGB‚1174.4983291625977( disk.nvme0c0n1.out‚49048.54736328125( disk.nvme1c1n1.out‚15900.80029296875' proc.memory.availableMB‚ 488.6796875# proc.memory.rssMB‚ 2840.85546875 network.sent‚ 1.3983023e+07* proc.memory.percent‚17.339205741882324 cpu‚26.121945523708003 proc.cpu.threads‚17UPyÈ“´‚(:ˆ ªÞôÒ–ä·Ì gpu.0.gpu‚91.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚143.403' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚47.801 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$¶wjµ‚(:ß ¹ÞôÒÐöÉË# disk.dm-0.out‚56542.41162109375( proc.memory.availableMB‚ 488.68359375 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375& disk.nvme1n1.out‚15901.87255859375 disk.dm-1.out‚8114.802734375# proc.memory.rssMB‚ 2840.85546875 cpu‚26.120901922528635 proc.cpu.threads‚17% disk./.usageGB‚1174.5063514709473' disk.nvme0c0n1.out‚49053.4736328125 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0$ memory_percent‚97.01731204986572* proc.memory.percent‚17.339205741882324 disk.rbd0.in‚0% disk.nvme0n1.out‚49053.4736328125 disk.nvme1n1.in‚0 network.sent‚ 1.4031592e+07 network.recv‚ 1.429563e+06) disk./.usagePercent‚76.50261204826232( disk.nvme1c1n1.out‚15901.87255859375 disk.md0.in‚0 disk.md0.out‚138.625»yŽž¶‚(:“ ¹ÞôÒŒ¹¹Ë gpu.0.gpu‚60.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚187.319' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚62.43966666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%8à2·- _wandbruntime‚5313  _runtime‚5313b ¹ñȸ ' lr-AdamW/pg1‚0.00018383997098347663 ' lr-AdamW/pg2‚0.00018383997098347663 trainer/global_step‚10249 # _timestamp‚1.78449183686032e+09 _runtime‚5313.436499888 _step‚828'B(Í¹Ç ' lr-AdamW/pg2‚0.00018383997098347663 trainer/global_step‚10249 # _timestamp‚1.78449183686032e+09 _runtime‚5313.436499888 _step‚828 ' lr-AdamW/pg1‚0.00018383997098347663¼§Ì¬ã2º- _wandbruntime‚5313  _runtime‚5313*F+ð'»êO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9814861002276 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9815200925598 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 _runtime‚5313.438814812 _step‚829 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79793450 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10134486439936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚783849526498816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9832548883456 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79793452 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚793682057863680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183361769 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124781568 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103568317 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79793452 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103558174 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚622630921274880 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27079607 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚785454923408916 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚793682416168448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103568317 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81398977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚632765556920832 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9832889669632 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚783849526498816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚785455280371598 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚775640079446000 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79793450 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9832548883456 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚793682416168448 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54319082 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183351624 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚632765407714816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10134611221504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚793682057863680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚783849508980224 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚622630945699328 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9832889669632 % _timestamp‚1.7844918368637686e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚783849508980224«iÝ·J¼ïO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚793682057863680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚783849508980224 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚622630945699328 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9832889669632 % _timestamp‚1.7844918368637686e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚783849508980224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9814861002276 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9815200925598 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 _runtime‚5313.438814812 _step‚829 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79793450 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10134486439936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚783849526498816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9832548883456 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79793452 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚793682057863680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183361769 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124781568 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103568317 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79793452 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103558174 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚622630921274880 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 TGDeviceStatsMonitor.on_train_batÆ·]Å« ch_start/inactive_split.small_pool.freed‚27079607 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚785454923408916 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚793682416168448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103568317 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81398977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚632765556920832 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9832889669632 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚783849526498816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚785455280371598 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚775640079446000 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79793450 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9832548883456 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚793682416168448 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54319082 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183351624 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚632765407714816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10134611221504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360½\ Æù2½- _wandbruntime‚5313  _runtime‚53135r£ú&¾ôM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103568049 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9833478794752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79800952 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183379160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79800954 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚5313.816431937 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚785529277714632 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚785529634677426 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 183369001 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54324161 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81406219 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10135440930304 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9815789225512 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚783923700749312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81406516 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 183379160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 183369001 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚793757179544064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚297 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9816129148946 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚632825238537728 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚295 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103578206 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚775713505528480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚632825387736576 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚622689822031872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10135565704704 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 % _timestamp‚1.7844918372408423e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103568049 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27082058 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79800954 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚793757537856000 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚783923718267904 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27082353 _step‚830@û%Òÿ&¿ùM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚622689797607424 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9833478794752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103578206 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚10249 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚783923718267904 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81406219 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10135440930304 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9815789225512 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚783923700749312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81406516 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 183379160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 183369001 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚793757179544064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚297 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9816129148946 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚632825238537728 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚295 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103578206 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚775713505528480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚632825387736576 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚622689822031872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10135565704704 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 % _timestamp‚1.7844918372408423e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103568049 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27082058 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79800954 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚793757537856000 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚783923718267904 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27082353 _step‚830 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103568049 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9833478794752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79800952 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183379160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79800954 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚5313.816431937 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚785529277714632 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚785529634677426 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 183369001 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54324161 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E¬â(2È- _wandbruntime‚5338  _runtime‚5338EÐdJwÉêO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚787559114953216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚625574762326528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54573132 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9861643990606 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54573130 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚635757406816256 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184222326 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚787559114953216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27206630 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80168550 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27206333 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80168550 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104053776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104063919 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81779463 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10182511319552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚5338.256121014 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 184232471 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚787559097434624 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9879076705280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚635757249221632 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚625574737902080 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9879417491456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9879417491456 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \HDevi(ò©yceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚787559097434624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚797438174139904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81779762 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9861304067284 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚797438532444672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80168552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 _step‚833 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚789172670597924 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚779311366530640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 trainer/global_step‚10299 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚297 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9879076705280 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10182644489728 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 % _timestamp‚1.7844918616797366e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720—)ôÃõ'ÊïO BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚797438174139904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81779762 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9861304067284 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚797438532444672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80168552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 _step‚833 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚789172670597924 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚779311366530640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 trainer/global_step‚10299 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚297 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9879076705280 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10182644489728 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 % _timestamp‚1.7844918616797366e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚787559114953216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚625574762326528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54573132 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9861643990606 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54573130 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚635757406816256 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184222326 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚787559114953216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27206630 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80168550 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27206333 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80168550 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104053776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104063919 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81779463 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10182511319552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚5338.256121014 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 184232471 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚787559097434624 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9879076705280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚635757249221632 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚625574737902080 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9879417491456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9879417491456 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚787559097434624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0ÁJÛ{b2Ë- _wandbruntime‚5338  _runtime‚5338^͇ú&ÌôM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚5338.621452865 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9862572213954 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104063651 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚779384792613120 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9862232290520 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184239703 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104073808 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚625633638659072 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27208793 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚789247381866434 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184239703 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81787004 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54578213 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184249862 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80176052 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚787633306722304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚797513295820288 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚779384809652480 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184249862 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 trainer/global_step‚10299 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27209083+ò]C ÍùM EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184249862 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 trainer/global_step‚10299 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27209083 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚5338.621452865 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9862572213954 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104063651 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚779384792613120 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9862232290520 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184239703 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104073808 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚625633638659072 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27208793 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚789247381866434 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 D>DeviceStaî…øë¼tsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184239703 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81787004 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54578213 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184249862 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80176052 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚787633306722304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚797513295820288 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚779384809652480 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434ÂÖBн2Î- _wandbruntime‚5338  _runtime‚5338àÛ[i Ï„ 2loss/SRqqt2/assignment_loss‚0.6821816564774195 2loss/FBt2/assignment_loss‚0.007204636558623652 /loss/FRt1/detection_loss‚1.3628393115512034 1loss/SRqqt2/detection_loss‚0.9650360921809038 _step‚835 0loss/FRt2/assignment_loss‚1.7777644100341392 2loss/FBt1/assignment_loss‚0.000562630609074688 1loss/SRqqt1/detection_loss‚0.6051598283538774 /loss/FBt2/detection_loss‚0.5265969019383192 %loss/total_loss‚8.237442945601842 _runtime‚5338.622129996 /loss/FRt2/detection_loss‚0.5693585129978601 trainer/global_step‚10299 0loss/FRt1/assignment_loss‚0.8478122421952544 2loss/SRqqt1/assignment_loss‚0.5273138048853494 /loss/FBt1/detection_loss‚0.3656129178198171 epoch‚12 $ _timestamp‚1.784491862047359e+09èýr£Ð‰ 2loss/FBt2/assignment_loss‚0.007204636558623652 /loss/FRt1/detection_loss‚1.3628393115512034 1loss/SRqqt2/detection_loss‚0.9650360921809038 _step‚835 0loss/FRt2/assignment_loss‚1.7777644100341392 2loss/FBt1/assignment_loss‚0.000562630609074688 1loss/SRqqt1/detection_loss‚0.6051598283538774 /loss/FBt2/detection_loss‚0.5265969019383192 %loss/total_loss‚8.237442945601842 _runtime‚5338.622129996 /loss/FRt2/detection_loss‚0.5693585129978601 trainer/global_step‚10299 0loss/FRt1/assignment_loss‚0.8478122421952544 2loss/SRqqt1/assignment_loss‚0.5273138048853494 /loss/FBt1/detection_loss‚0.3656129178198171 epoch‚12 $ _timestamp‚1.784491862047359e+09 2loss/SRqqt2/assignment_loss‚0.6821816564774195Ã{¦eUoÑ‚(:ä ×ÞôÒž§ÅË network.sent‚ 1.4098877e+07) disk./.usagePercent‚76.48666250144711 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚15961.6494140625 disk.nvme1n1.in‚0% disk.nvme1n1.out‚15961.6494140625# disk.dm-0.out‚56794.16650390625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0" disk.dm-1.out‚8115.03564453125$ memory_percent‚97.01964855194092# proc.memory.rssMB‚ 2840.85546875 network.recv‚ 1.437526e+06' disk.nvme0c0n1.out‚49247.3115234375* proc.memory.percent‚17.339205741882324 proc.cpu.threads‚17% disk./.usageGB‚1174.2614860534668 disk.rbd0.out‚ 0.4697265625% disk.nvme0n1.out‚49247.3115234375 disk.md0.out‚ 139.37109375( proc.memory.availableMB‚ 488.30078125 cpu‚26.118917384188894BéèûœÒ‚(:‘ ×ÞôÒýÎóË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚264.035' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.01166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø Ü`Ó‚(:Õ æÞôÒË´ÓË disk.md0.in‚0" disk.dm-1.out‚8115.17626953125 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625# proc.memory.rssMB‚ 2840.85546875* proc.memory.percent‚17.339205741882324% disk./.usageGB‚1174.2621002197266 disk.nvme1c1n1.in‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 4.3359375" disk.nvme0c0n1.out‚ 49248.46875 disk.nvme0n1.out‚ 49248.46875 proc.cpu.threads‚17 network.sent‚ 1.4163177e+07& disk.nvme1c1n1.out‚15962.251953125 disk.md0.out‚ 139.7109375# disk.dm-0.out‚56795.05712890625 disk.nvme0n1.in‚ 4.3359375 network.recv‚ 1.442181e+06 disk.nvme1n1.in‚0 disk.dm-0.in‚ 4.3359375$ memory_percent‚97.01826572418213( proc.memory.availableMB‚ 488.52734375 cpu‚26.118537461801896' disk./.usagePercent‚76.486702505763$ disk.nvme1n1.out‚15962.251953125û1žÔ‚(:“ æÞôÒã÷ˆÌ gpu.0.gpu‚72.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚179.204' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚59.73466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾…‰2Õ- _wandbruntime‚5362  _runtime‚5362êZ€‰ÊÖÄ ' lr-AdamW/pg1‚0.00018325296330132823 ' lr-AdamW/pg2‚0.00018325296330132823 trainer/global_step‚10349 % _timestamp‚1.7844918860006344e+09 _runtime‚5362.577650927 _step‚836¿­xÏ×É ' lr-AdamW/pg1‚0.00018325296330132823 ' lr-AdamW/pg2‚0.00018325296330132823 trainer/global_step‚10349 % _timestamp‚1.7844918860006344e+09 _runtime‚5362.577650927 _step‚836ÄJÉ÷2Ø- _wandbruntime‚5362  _runtime‚5362Wåðï'ÙéO [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚791268703407616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10230636947968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10349 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚638749064598528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚791268685889024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚792890774749614 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9907747132292 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54827182 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚801194648720896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9925945313280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82160935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 $ _timestamp‚1.784491886005078e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚791268685889024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚792890417786932 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚801194290416128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80543650 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104559521 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9925945313280 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚801194648720896 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0ÁRE ô'ÚîO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104559521 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚638749215901696 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 _step‚837 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80543652 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9925945313280 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚801194648720896 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚791268703407616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10230636947968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10349 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚638749064598528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚791268685889024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚792890774749614 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9907747132292 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54827182 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚801194648720896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9925945313280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82160935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 $ _timestamp‚1.784491886005078e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚791268685889024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚792890417786932 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚801194290416128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80543650 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104559521 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82168518 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚801269770408448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185120564 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚783056096737120 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104569410 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚801269770408448 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 % _timestamp‚1.7844918863617399e+09 BŒ9015278962 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9926875231744 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9908675355528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚792965129055442 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80551154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10231599193088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9926534438400 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 trainer/global_step‚10349 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9926534438400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27336257 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185110405 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54832263 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9926875231744 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104559253 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚628577455286272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27336547 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104569410 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚791342895176704 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚791342895176704 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚791342877658112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720£.”¿ÿ&ÝùM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27336547 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104569410 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚791342895176704 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚791342895176704 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚791342877658112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80551152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚792964772092648 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82168518 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚801269770408448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185120564 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚783056096737120 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104569410 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚801269770408448 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 % _timestamp‚1.7844918863617399e+09 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚792965129055442 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80551154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10231599193088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9926534438400 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 trainer/global_step‚10349 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9926534438400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27336257 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185110405 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54832263 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9926875231744 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104559253 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚628577455286272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 179347968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0Æë¤ûb2Þ- _wandbruntime‚5362  _runtime‚5362U;$Ï ß† 0loss/FRt1/assignment_loss‚0.8457231528845875 2loss/SRqqt1/assignment_loss‚0.6031987625043786 %loss/total_loss‚9.005628618012548 /loss/FRt2/detection_loss‚0.5473246275752439 1loss/SRqqt1/detection_loss‚0.6481721299466956 /loss/FBt1/detection_loss‚0.3710027962896815 /loss/FBt2/detection_loss‚1.1164430268399883 epoch‚12 % _timestamp‚1.7844918863631217e+09 _step‚839 0loss/FRt2/assignment_loss‚1.7933681983696788 3loss/FBt2/assignment_loss‚3.699036606263917e-05 trainer/global_step‚10349 _runtime‚5362.937788056 2loss/SRqqt2/assignment_loss‚0.6437713862315728 3loss/FBt1/assignment_loss‚0.0017910650970606758 .loss/FRt1/detection_loss‚1.476594858627525 1loss/SRqqt2/detection_loss‚0.9582016232800729òê+à‹ 2loss/SRqqt2/assignment_loss‚0.6437713862315728 3loss/FBt1/assignment_loss‚0.0017910650970606758 .loss/FRt1/detection_loss‚1.476594858627525 1loss/SRqqt2/detection_loss‚0.9582016232800729 0loss/FRt1/assignment_loss‚0.8457231528845875 2loss/SRqqt1/assignment_loss‚0.6031987625043786 %loss/total_loss‚9.005628618012548 /loss/FRt2/detection_loss‚0.5473246275752439 1loss/SRqqt1/detection_loss‚0.6481721299466956 /loss/FBt1/detection_loss‚0.3710027962896815 /loss/FBt2/detection_loss‚1.1164430268399883 epoch‚12 % _timestamp‚1.7844918863631217e+09 _step‚839 0loss/FRt2/assignment_loss‚1.7933681983696788 3loss/FBt2/assignment_loss‚3.699036606263917e-05 trainer/global_step‚10349 _runtime‚5362.937788056ÇXß„má‚(:â õÞôÒ‰Ì) disk./.usagePercent‚76.48665653807083" disk.dm-1.out‚8115.30908203125 disk.rbd0.out‚ 0.4697265625% disk.nvme0n1.out‚49266.9111328125$ memory_percent‚97.02036380767822# proc.memory.rssMB‚ 2840.85546875 cpu‚26.11936323673273% disk./.usageGB‚1174.2613945007324 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚15967.40380859375 disk.nvme1n1.in‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 488.18359375 network.sent‚ 1.4178942e+07 disk.md0.out‚ 140.12890625 disk.dm-0.out‚56817.62109375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375' disk.nvme0c0n1.out‚49266.9111328125 proc.cpu.threads‚17& disk.nvme1n1.out‚15967.40380859375 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375* proc.memory.percent‚17.339205741882324 network.recv‚ 1.445165e+06`ÁSžâ‚(:“ õÞôÒÛç¾Ì gpu.0.gpu‚68.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚233.815' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚77.93833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éiã‚(:Þ „ßôÒÉ»þË disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚49267.97802734375$ disk.nvme1n1.out‚15975.943359375 disk.dm-1.in‚0 cpu‚26.118894077928385 disk.nvme0c0n1.in‚ 4.3359375( disk.nvme0c0n1.out‚49267.97802734375& disk.nvme1c1n1.out‚15975.943359375 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 disk.rbd0.out‚ 0.4697265625$ memory_percent‚97.02000617980957 disk.nvme1c1n1.in‚0 disk.dm-0.out‚56826.51953125 disk.rbd0.in‚0# proc.memory.rssMB‚ 2840.85546875* proc.memory.percent‚17.339205741882324% disk./.usageGB‚1174.3199157714844 disk.md0.out‚ 140.390625" disk.dm-1.out‚8115.45361328125' proc.memory.availableMB‚ 488.2421875 proc.cpu.threads‚17 network.sent‚ 1.4230364e+07 network.recv‚ 1.449753e+06) disk./.usagePercent‚76.49046837788532eˆÒžä‚(:“ „ßôÒÏÔ´Ì gpu.0.gpu‚90.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚127.951' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚42.65033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éÙ˜í2å- _wandbruntime‚5387  _runtime‚5387Â}7ÊæÄ ' lr-AdamW/pg2‚0.00018295761126221432 trainer/global_step‚10399 % _timestamp‚1.7844919108780599e+09 _runtime‚5387.453045086 _step‚840 ' lr-AdamW/pg1‚0.00018295761126221432¼‰”ÛÏçÉ ' lr-AdamW/pg2‚0.00018295761126221432 trainer/global_step‚10399 % _timestamp‚1.7844919108780599e+09 _runtime‚5387.453045086 _step‚840 ' lr-AdamW/pg1‚0.00018295761126221432Èk-Ô“2è- _wandbruntime‚5387  _runtime‚5387QB›Oð'éêO G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚786653974778640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9972132348928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚297 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105044980 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7844919108792126e+09 _runtime‚5387.454280487 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80918752 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚794978291862016 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚631462371156480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚786653991818000 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82541932 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚796608164975940 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 trainer/global_step‚10399 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚804950764997120 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185973875 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚641741035501568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 185963730 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55081232 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚794978291862016 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚796608521938622 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚804950764997120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚804950406692352 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9972473135104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497472 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10278639920640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105055123 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚804950406692352 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9954190197300 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105044980 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9972473135104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9972132348928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80918750 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27460999 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚641740880004096 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚794978274343424 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82542231 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚786653991818000 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82541932 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚796608164975940 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 trainer/global_step‚10399 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚804950764997120 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185973875 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚641741035501568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 185963730 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55081232 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 ]IDeviceStat‹\Þ7sMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚794978291862016 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚796608521938622 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚804950764997120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚804950406692352 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9972473135104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497472 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10278639920640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 179347968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105055123 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚804950406692352 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9954190197300 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105044980 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9972473135104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9972132348928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80918750 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27460999 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚641740880004096 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚794978274343424 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82542231 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚786653974778640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9972132348928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚297ÉZMÙ2ë- _wandbruntime‚5387  _runtime‚5387\„ôú&ìôM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚641800717458432 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185991266 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9955458343970 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚795052483631104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105054855 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9973403053568 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27463500 trainer/global_step‚10399 % _timestamp‚1.7844919112534623e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82549512 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚805025886684672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚805025886684672 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚795052466112512 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105054855 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55086313 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚631521271913472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚786727417900480 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9955118420536 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185991266 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80926254 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚795052466112512 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9973062260224 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105065012 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82549813 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚796682519281656 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚786727400861120 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 _step‚842 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105065012 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚805025528372736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9973062260224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10279469969408 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5387.828602288 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185981107 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80926252 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27463201 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚796682876244450 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚631521247489024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80926254 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚299 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80926252 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592ˆâguÿ&íùM K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105054855 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9973403053568 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27463500 trainer/global_step‚10399 % _timestamp‚1.7844919112534623e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82549512 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚805025886684672 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚805025886684672 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚795052466112512 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105054855 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55086313 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚631521271913472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚786727417900480 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9955118420536 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185991266 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80926254 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚795052466112512 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9973062260224 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105065012 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82549813 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚796682519281656 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚786727400861120 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 _step‚842 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105065012 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚805025528372736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9973062260224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10279469969408 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5387.828602288 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185981107 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80926252 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27463201 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚796682876244450 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚631521247489024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80926254 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚299 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80926252 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚641800717458432 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185991266 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9955458343970 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚795052483631104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260ʃÃæ2î- _wandbruntime‚5387  _runtime‚5387…äbï‚ 1loss/FBt2/assignment_loss‚0.03028565985523703 1loss/SRqqt1/detection_loss‚0.5702373834010113 /loss/FBt2/detection_loss‚0.3954724830408127 0loss/SRqqt2/assignment_loss‚0.49601206378457 /loss/FRt1/detection_loss‚1.3788620581107445 1loss/SRqqt2/detection_loss‚0.9247963870193359 _runtime‚5387.829354989 0loss/FRt1/assignment_loss‚0.6913437791977929 1loss/FBt1/assignment_loss‚0.00913446747513482 0loss/FBt1/detection_loss‚0.39687305443327536 %loss/total_loss‚7.720997829185437 /loss/FRt2/assignment_loss‚1.856698034914778 /loss/FRt2/detection_loss‚0.5134956328591433 epoch‚12 trainer/global_step‚10399 % _timestamp‚1.7844919112545464e+09 _step‚843 3loss/SRqqt1/assignment_loss‚0.45778682509360186î\Ž ð‡ 0loss/SRqqt2/assignment_loss‚0.49601206378457 /loss/FRt1/detection_loss‚1.3788620581107445 1loss/SRqqt2/detection_loss‚0.9247963870193359 _runtime‚5387.829354989 0loss/FRt1/assignment_loss‚0.6913437791977929 1loss/FBt1/assignment_loss‚0.00913446747513482 0loss/FBt1/detection_loss‚0.39687305443327536 %loss/total_loss‚7.720997829185437 /loss/FRt2/assignment_loss‚1.856698034914778 /loss/FRt2/detection_loss‚0.5134956328591433 epoch‚12 trainer/global_step‚10399 % _timestamp‚1.7844919112545464e+09 _step‚843 3loss/SRqqt1/assignment_loss‚0.45778682509360186 1loss/FBt2/assignment_loss‚0.03028565985523703 1loss/SRqqt1/detection_loss‚0.5702373834010113 /loss/FBt2/detection_loss‚0.3954724830408127ËÉÝEjlñ‚(:á “ßôÒ²ÜÌ' disk.nvme1c1n1.out‚16027.8916015625 network.recv‚ 1.452309e+06 disk.md0.in‚0 disk.md0.out‚ 140.67578125& disk.nvme0n1.out‚49271.18798828125 disk.nvme1n1.in‚0$ memory_percent‚97.02725410461426# proc.memory.rssMB‚ 2842.04296875) proc.memory.percent‚17.34645366668701% disk./.usageGB‚1174.3193588256836" disk.dm-1.out‚8115.69775390625 disk.nvme0n1.in‚ 4.3359375% disk.nvme1n1.out‚16027.8916015625 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0' proc.memory.availableMB‚ 487.0546875 cpu‚26.12031804310725 proc.cpu.threads‚17( disk.nvme0c0n1.out‚49271.18798828125 network.sent‚ 1.4245958e+07) disk./.usagePercent‚76.49043210067961! disk.dm-0.out‚56880.798828125 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375¹ßKžò‚(:“ “ßôÒ»ÊÀÌ gpu.0.gpu‚51.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚118.252' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.41733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XÝé¢2ó- _wandbruntime‚5413  _runtime‚5413"™S"Î ô÷z 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9946145415306091 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21443083882331848 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20920585095882416 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9953995943069458 +particle/sensitivity‚0.6406733393669128 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9642857313156128 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6338810324668884 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5957246422767639 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 6 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.631944477558136 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4449999928474426 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7378920912742615 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23553717136383057 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_3_of_5‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21185848116874695 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21890632808208466 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7603116035461426 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5856805443763733 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23553717136383057 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.760827898979187 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04001031816005707 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.27272728085517883 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9962481260299683 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14687499403953552 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 jet/accuracy_4_of_5‚0 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.040646784007549286 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.22721394896507263 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8500000238418579 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978741407394409 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.23242153227329254 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38641229271888733 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/2FRt*SRqq![ùç¯1t1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_4‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9972051978111267 *jet/accuracy_2_of_2‚0.1594267636537552 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29239746928215027 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6688386797904968 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7569023966789246 *jet/accuracy_2_of_3‚0.5706511735916138 'particle/f_score‚0.6493123769760132 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.29745498299598694 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912552833557129 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45764270424842834 jet/accuracy_1_of_5‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6818987131118774 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.038350027054548264 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24832622706890106 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3194444477558136 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.996170699596405 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8513541221618652 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8456140756607056 _step‚844 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.46529293060302734 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.27272728085517883 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7991830110549927 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6499921679496765 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1293504238128662 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9412009716033936 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6333929896354675 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.476703017950058 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 *jet/accuracy_1_of_1‚0.3345763683319092 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 )loss/val_total_loss‚8.066710188205947 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12544527649879456 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9902447462081909 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6477484107017517 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8787729144096375 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1704098880290985 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_2_of_2‚1 (particle/accuracy‚0.8514152765274048 _runtime‚5413.647972713 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6521929502487183 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5947092175483704 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7457531094551086 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7476298213005066 /particle/accuracy_4_of_4‚0.9360352754592896 *jet/accuracy_1_of_2‚0.7838334441184998 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8405389189720154 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7508546113967896 jet/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19857797026634216 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1964419186115265 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6978978514671326 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24873630702495575 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8500000238418579 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2093532830476761 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.654338538646698 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19099850952625275 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7441403865814209 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.739033043384552 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19548211991786957 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.689968466758728 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5155286192893982 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8184210062026978 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38964158296585083 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4128788113594055 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7391172051429749 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1995590776205063 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9980574250221252 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9957055449485779 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6842189431190491 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02885960042476654 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2462948113679886 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.38271602988243103 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.873318612575531 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6576968431472778 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6887989640235901 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9937929511070251 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05098050460219383 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03810574486851692 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8513541221618652 *jet/accuracy_1_of_3‚0.9258933067321777 trainer/global_step‚10412 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6754240989685059 7validation_average_jet_accuracy‚0.37629789113998413 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7314528226852417 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.679286003112793 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6097508668899536 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9885352849960327 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9935373663902283 7"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.74440598487854 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32959309220314026 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2132311314344406 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11321798712015152 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8423076272010803 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 jet/accuracy_2_of_5‚1 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998833417892456 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6749725341796875 particle/accuracy_5_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6823099255561829 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2901618182659149 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03689301759004593 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6521929502487183 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1795773208141327 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.07462610304355621 +jet/accuracy_4_of_4‚0.01666666567325592 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7707157135009766 )jet/accuracy_1_of_4‚0.974877119064331 +jet/accuracy_3_of_4‚0.08282280713319778 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8309523463249207 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9964585900306702 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03052854910492897 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_2_of_4‚0.8075942993164062 particle/accuracy_2_of_4‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.987900972366333 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9104451537132263 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4713653326034546 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9804894328117371 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5286458730697632 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7004828453063965 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8011969327926636 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9213560223579407 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46382004022598267 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1881571263074875 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.022078322246670723 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9914529323577881 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14181554317474365 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7666749358177185 particle/accuracy_3_of_3‚1 +particle/specificity‚0.9090451002120972 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9970735907554626 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 epoch‚12 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04212565720081329 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0714285746216774 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05793081596493721 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02885960042476654 particle/accuracy_1_of_1‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0714285746216774 # _timestamp‚1.78449193707252e+09 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6549830436706543—p"ù‚=õüz & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7378920912742615 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23553717136383057 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_3_of_5‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21185848116874695 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21890632808208466 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7603116035461426 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5856805443763733 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23553717136383057 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.760827898979187 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04001031816005707 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.27272728085517883 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9962481260299683 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14687499403953552 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 jet/accuracy_4_of_5‚0 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.040646784007549286 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.22721394896507263 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8500000238418579 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978741407394409 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.23242153227329254 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38641229271888733 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_4‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9972051978111267 *jet/accuracy_2_of_2‚0.1594267636537552 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29239746928215027 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6688386797904968 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7569023966789246 *jet/accuracy_2_of_3‚0.5706511735916138 'particle/f_score‚0.6493123769760132 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.29745498299598694 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912552833557129 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45764270424842834 jet/accuracy_1_of_5‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6818987131118774 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.038350027054548264 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24832622706890106 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3194444477558136 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.996170699596405 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8513541221618652 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8456140756607056 _step‚844 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.46529293060302734 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.27272728085517883 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7991830110549927 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6499921679496765 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1293504238128662 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9412009716033936 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6333929896354675 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.476703017950058 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 *jet/accuracy_1_of_1‚0.3345763683319092 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 )loss/val_total_loss‚8.066710188205947 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12544527649879456 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9902447462081909 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6477484107017517 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8787729144096375 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1704098880290985 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_2_of_2‚1 (particle/accuracy‚0.8514152765274048 _runtime‚5413.647972713 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6521929502487183 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5947092175483704 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7457531094551086 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7476298213005066 /particle/accuracy_4_of_4‚0.9360352754592896 *jet/accuracy_1_of_2‚0.7838334441184998 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8405389189720154 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7508546113967896 jet/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19857797026634216 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1964419186115265 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6978978514671326 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24873630702495575 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8500000238418579 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2093532830476761 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.654338538646698 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19099850952625275 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7441403865814209 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.739033043384552 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19548211991786957 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.689968466758728 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5155286192893982 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8184210062026978 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38964158296585083 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4128788113594055 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7391172051429749 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1995590776205063 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9980574250221252 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9957055449485779 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6842189431190491 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02885960042476654 ,"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.125 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2462948113679886 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.38271602988243103 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.873318612575531 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6576968431472778 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6887989640235901 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9937929511070251 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05098050460219383 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03810574486851692 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8513541221618652 *jet/accuracy_1_of_3‚0.9258933067321777 trainer/global_step‚10412 ,"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.125 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6754240989685059 7validation_average_jet_accuracy‚0.37629789113998413 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7314528226852417 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.679286003112793 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6097508668899536 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9885352849960327 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9935373663902283 7"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.74440598487854 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32959309220314026 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2132311314344406 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11321798712015152 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8423076272010803 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 jet/accuracy_2_of_5‚1 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998833417892456 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6749725341796875 particle/accuracy_5_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6823099255561829 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2901618182659149 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03689301759004593 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6521929502487183 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1795773208141327 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.07462610304355621 +jet/accuracy_4_of_4‚0.01666666567325592 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7707157135009766 )jet/accuracy_1_of_4‚0.974877119064331 +jet/accuracy_3_of_4‚0.08282280713319778 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8309523463249207 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9964585900306702 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03052854910492897 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_2_of_4‚0.8075942993164062 particle/accuracy_2_of_4‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.987900972366333 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9104451537132263 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4713653326034546 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9804894328117371 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5286458730697632 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7004828453063965 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8011969327926636 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9213560223579407 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46382004022598267 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1881571263074875 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.022078322246670723 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9914529323577881 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14181554317474365 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7666749358177185 particle/accuracy_3_of_3‚1 +particle/specificity‚0.9090451002120972 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9970735907554626 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 epoch‚12 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04212565720081329 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0714285746216774 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05793081596493721 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02885960042476654 particle/accuracy_1_of_1‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0714285746216774 # _timestamp‚1.78449193707252e+09 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6549830436706543 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9946145415306091 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21443083882331848 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20920585095882416 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9953995943069458 +particle/sensitivity‚0.6406733393669128 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9642857313156128 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6338810324668884 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5957246422767639 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 6 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.631944477558136 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4449999928474426ÌU¶Iq_ö‚(:Ô ¢ßôÒö©éË disk.dm-1.in‚0" disk.dm-1.out‚8361.20166015625 disk.rbd0.in‚0( disk.nvme0c0n1.out‚49613.57763671875 disk.nvme1n1.in‚0 network.recv‚ 1.456616e+06) disk./.usagePercent‚76.50593663053974 disk.md0.in‚0! disk.dm-0.out‚57370.677734375 disk.rbd0.out‚ 0.4697265625 disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚49613.57763671875# disk.nvme1c1n1.out‚ 16421.765625 disk.md0.out‚ 141.09375% proc.memory.availableMB‚ 739.21875* proc.memory.percent‚17.352581024169922 cpu‚26.125560675992503 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0! disk.nvme1n1.out‚ 16421.765625$ memory_percent‚95.48816680908203! proc.memory.rssMB‚ 2843.046875 proc.cpu.threads‚17 network.sent‚ 1.4307541e+07% disk./.usageGB‚1174.5573921203613 disk.dm-0.in‚ 4.3359375Òfby÷‚(:„ ¢ßôÒþ©ÝË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚241.38' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚80.46 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l>Wiø‚(:Þ ±ßôÒ“¥ÈË disk.md0.out‚ 141.3984375# disk.dm-0.out‚57378.42626953125! disk.dm-1.out‚8361.3779296875 disk.nvme1c1n1.in‚0 network.sent‚ 1.4351858e+07 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375& disk.nvme0c0n1.out‚49621.509765625 disk.nvme0n1.in‚ 4.3359375$ disk.nvme0n1.out‚49621.509765625! proc.memory.rssMB‚ 2843.046875 cpu‚26.126411293565134 network.recv‚ 1.460659e+06$ disk./.usageGB‚1174.557674407959 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0& disk.nvme1n1.out‚16422.43017578125$ memory_percent‚95.48811912536621* proc.memory.percent‚17.352581024169922) disk./.usagePercent‚76.50595501761661 disk.rbd0.in‚0( disk.nvme1c1n1.out‚16422.43017578125 disk.nvme1n1.in‚0' proc.memory.availableMB‚ 739.2265625 proc.cpu.threads‚17f(–žù‚(:“ ±ßôÒÍ‹Ì gpu.0.gpu‚90.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚254.497' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.83233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z÷ÄH2ú- _wandbruntime‚5432  _runtime‚5432„‰ÜÿÊûÄ ' lr-AdamW/pg1‚0.00018266103339205378 ' lr-AdamW/pg2‚0.00018266103339205378 trainer/global_step‚10449 % _timestamp‚1.7844919562579627e+09 _runtime‚5432.832938574 _step‚8454E¹;ÏüÉ ' lr-AdamW/pg1‚0.00018266103339205378 ' lr-AdamW/pg2‚0.00018266103339205378 trainer/global_step‚10449 % _timestamp‚1.7844919562579627e+09 _runtime‚5432.832938574 _step‚845Í6®R2ý- _wandbruntime‚5432  _runtime‚5432å÷ÜþîO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105704152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚800634853451264 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚635999098299904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚810678029677056 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187249311 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚802267830294940 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_Á‘³Ìd#pool.peak‚ 11712753664 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83209023 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55521760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚792242367105840 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚800634853451264 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚646350300114944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81535014 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚810678387981824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚294 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105714295 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497472 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚800634870969856 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10025123265778 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27687263 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚802267473332258 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 # _timestamp‚1.78449195625894e+09 _runtime‚5432.835182168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 _step‚846 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚810678029677056 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81535016 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 187239166 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10043517011968 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚792242350066480 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 trainer/global_step‚10449 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0&o ºù'ÿóO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 187239166 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10043517011968 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚792242350066480 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 trainer/global_step‚10449 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105704152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚800634853451264 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚635999098299904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚810678029677056 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187249311 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚802267830294940 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83209023 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55521760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚792242367105840 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚800634853451264 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚646350300114944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81535014 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚810678387981824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚294 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105714295 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497472 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚800634870969856 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10025123265778 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27687263 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚802267473332258 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 # _timestamp‚1.78449195625894e+09 _runtime‚5432.835182168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 _step‚846 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚810678029677056 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81535016 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ÎFå2€- _wandbruntime‚5433  _runtime‚5433|#q('úM ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10044446930432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 _step‚847 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚810753509669376 trainer/global_step‚10449 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚810753509669376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27690067 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10044106137088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187256543 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55526843 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 187256543 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81542516 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚800709045220352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚792315776148960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 % _timestamp‚1.7844919566342194e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10044106137088 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187266702 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10026051489014 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105714027 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105724184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10352317199872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81542518 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚800709062738944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚800709045220352 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10044446930432 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚636057950208000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10026391412448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81542516 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83216624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55526841 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚646410291832320 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚636057974632448 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚810753151357440WR§ÿN ‚ÿM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81542516 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83216624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55526841 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚646410291832320 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚636057974632448 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚810753151357440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10044446930432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 _step‚847 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚810753509669376 trainer/global_step‚10449 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚810753509669376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27690067 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10044106137088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freved‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187256543 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55526843 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 187256543 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81542516 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚800709045220352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚792315776148960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 % _timestamp‚1.7844919566342194e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10044106137088 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187266702 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10026051489014 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105714027 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105724184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10352317199872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81542518 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚800709062738944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚800709045220352 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10044446930432 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚636057950208000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10026391412448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592Ï2’O2ƒ- _wandbruntime‚5433  _runtime‚5433"Õæô„ˆ trainer/global_step‚10449 _runtime‚5433.209955169 4loss/FBt1/assignment_loss‚0.00015545030389384773 /loss/FRt2/detection_loss‚0.5639000694111113 0loss/FRt1/assignment_loss‚0.9203930232259963 /loss/FBt1/detection_loss‚0.5564822667812998 _step‚848 2loss/SRqqt1/assignment_loss‚0.5840558878842926 3loss/FBt2/assignment_loss‚0.0008178137680834837 1loss/SRqqt1/detection_loss‚0.6225976292396189 1loss/SRqqt2/detection_loss‚0.9059706250180966 epoch‚13 % _timestamp‚1.7844919566352687e+09 0loss/FRt2/assignment_loss‚1.8541584234892554 2loss/SRqqt2/assignment_loss‚0.5351153262037035 /loss/FRt1/detection_loss‚1.3870747964161592 /loss/FBt2/detection_loss‚0.5254820586359572 %loss/total_loss‚8.456203370377468±ß<… 2loss/SRqqt1/assignment_loss‚0.5840558878842926 3loss/FBt2/assignment_loss‚0.0008178137680834837 1loss/SRqqt1/detection_loss‚0.6225976292396189 1loss/SRqqt2/detection_loss‚0.9059706250180966 epoch‚13 % _timestamp‚1.7844919566352687e+09 0loss/FRt2/assignment_loss‚1.8541584234892554 2loss/SRqqt2/assignment_loss‚0.5351153262037035 /loss/FRt1/detection_loss‚1.3870747964161592 /loss/FBt2/detection_loss‚0.5254820586359572 %loss/total_loss‚8.456203370377468 trainer/global_step‚10449 _runtime‚5433.209955169 4loss/FBt1/assignment_loss‚0.00015545030389384773 /loss/FRt2/detection_loss‚0.5639000694111113 0loss/FRt1/assignment_loss‚0.9203930232259963 /loss/FBt1/detection_loss‚0.5564822667812998 _step‚848ÐòÌÒb†‚(:× ÀßôÒ¼¯ÞË( disk./.usagePercent‚76.5059443332341# disk.dm-0.out‚57379.12548828125 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1n1.in‚0 network.sent‚ 1.4357742e+07 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚16668.49658203125! proc.memory.rssMB‚ 2843.046875* proc.memory.percent‚17.352581024169922 cpu‚26.12821465169869 network.recv‚ 1.464201e+06 disk.md0.out‚141.8125 disk.dm-1.in‚0! disk.dm-1.out‚8606.9248046875 disk.rbd0.in‚0$ disk.nvme0n1.out‚49622.580078125# memory_percent‚95.4885482788086% proc.memory.availableMB‚ 739.15625% disk./.usageGB‚1174.5575103759766 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 disk.rbd0.out‚ 0.4697265625 disk.nvme0n1.in‚ 4.3359375& disk.nvme1n1.out‚16668.49658203125 proc.cpu.threads‚17& disk.nvme0c0n1.out‚49622.580078125;Â/“‡‚(:ˆ ÀßôÒáÐÀÌ gpu.0.gpu‚56.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚259.536' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚86.512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èh­¡2ˆ- _wandbruntime‚5458  _runtime‚5458ßAøÊ‰Ä ' lr-AdamW/pg1‚0.00018236323444114318 ' lr-AdamW/pg2‚0.00018236323444114318 trainer/global_step‚10499 % _timestamp‚1.7844919815690317e+09 _runtime‚5458.143990444 _step‚849»,][ÏŠÉ _step‚849 ' lr-AdamW/pg1‚0.00018236323444114318 ' lr-AdamW/pg2‚0.00018236323444114318 trainer/global_step‚10499 % _timestamp‚1.7844919815690317e+09 _runtime‚5458.143990444Ñœ4ë2‹- _wandbruntime‚5458  _runtime‚5458À¦žö'ŒðO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚804344441905664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81910116 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚649342197093376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚638942914927104 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚638942890502656 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83588855 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10399282166272 _step‚850 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10090044833792 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚795913654190480 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188120013 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚814434145953280 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83589146 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10071906254108 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚814434504258048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚805985577483948 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚805985220521266 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 % _timestamp‚1.7844919815697105e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81910114 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚804344459424256 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚814434145953280 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚814434504258048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81910114 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27813334 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚804344459424256 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106209897 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188120013 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚814434145953280 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83589146 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10071906254108 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚814434504258048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚805985577483948 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚805985220521266 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 % _timestamp‚1.7844919815697105e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81910114 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚804344459424256 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚814434145953280 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚814434504258048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81910114 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27813334 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚804344459424256 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106209897 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446Ò†Ÿ42Ž- _wandbruntime‚5458  _runtime‚5458šÄ+úM OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106209629 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚795987097312320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚649402037161984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81917616 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81917618 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83596466 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10400245902336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚804418651193344 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10090974752256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10090974752256 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚806059931789776 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 JBDeviceStatsÄ5³Monitor.on_train_batch_end/segment.large_pool.allocated‚675 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188127245 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27815575 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚649401881671680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 _step‚851 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚804418633674752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚804418651193344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚639001791259648 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10400114836480 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106219786 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27815866 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10072834477456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚804418633674752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚814509267633664 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10072494554022 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚814509625945600 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10090633958912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188137404 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚10499 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚795987080272960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106219786 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188137404 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81917616 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81917618 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10090633958912 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83596759 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360hÕÎ'ÿM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188137404 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚10499 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚795987080272960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106219786 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188137404 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81917616 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81917618 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10090633958912 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83596759 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106209629 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚795987097312320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚649402037161984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81917616 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81917618 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83596466 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10400245902336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚804418651193344 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10090974752256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10090974752256 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚806059931789776 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188127245 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27815575 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚649401881671680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 _step‚851 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚804418633674752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚804418651193344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚639001791259648 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10400114836480 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106219786 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27815866 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10072834477456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚804418633674752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚814509267633664 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10072494554022 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚814509625945600 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10090633958912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520Ó˜ö­2‘- _wandbruntime‚5458  _runtime‚5458œ Î ’… 0loss/FBt1/detection_loss‚0.43273064398220673 0loss/FRt1/assignment_loss‚0.7348879761846641 2loss/SRqqt2/assignment_loss‚0.6529119558203093 /loss/FRt1/detection_loss‚1.2768723177465908 1loss/SRqqt1/detection_loss‚0.6420908506891081 trainer/global_step‚10499 _runtime‚5458.833446446 2loss/SRqqt1/assignment_loss‚0.5371353268253803 /loss/FRt2/detection_loss‚0.5643147615287333 1loss/SRqqt2/detection_loss‚1.0084676854428836 % _timestamp‚1.7844919822585254e+09 3loss/FBt1/assignment_loss‚0.0015958208349148245 /loss/FBt2/detection_loss‚0.4708572970703244 %loss/total_loss‚8.033279587232315 epoch‚13 _step‚852 /loss/FRt2/assignment_loss‚1.710093764939608 1loss/FBt2/assignment_loss‚0.00132118616759109Ú$“Š /loss/FBt2/detection_loss‚0.4708572970703244 %loss/total_loss‚8.033279587232315 epoch‚13 _step‚852 /loss/FRt2/assignment_loss‚1.710093764939608 1loss/FBt2/assignment_loss‚0.00132118616759109 0loss/FBt1/detection_loss‚0.43273064398220673 0loss/FRt1/assignment_loss‚0.7348879761846641 2loss/SRqqt2/assignment_loss‚0.6529119558203093 /loss/FRt1/detection_loss‚1.2768723177465908 1loss/SRqqt1/detection_loss‚0.6420908506891081 trainer/global_step‚10499 _runtime‚5458.833446446 2loss/SRqqt1/assignment_loss‚0.5371353268253803 /loss/FRt2/detection_loss‚0.5643147615287333 1loss/SRqqt2/detection_loss‚1.0084676854428836 % _timestamp‚1.7844919822585254e+09 3loss/FBt1/assignment_loss‚0.0015958208349148245Ô˜£îk”‚(:à ÏßôÒ¯êÚË disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0 disk.md0.in‚0" disk.dm-1.out‚8607.08642578125$ memory_percent‚95.49148082733154! proc.memory.rssMB‚ 2843.046875% disk./.usageGB‚1174.5550155639648 disk.nvme0n1.in‚ 4.3359375' disk.nvme1c1n1.out‚16673.7021484375 disk.nvme1n1.in‚0% disk.nvme1n1.out‚16673.7021484375 disk.dm-0.in‚ 4.3359375( proc.memory.availableMB‚ 738.67578125* proc.memory.percent‚17.352581024169922 network.sent‚ 1.44201e+07 disk.md0.out‚ 142.1484375" disk.dm-0.out‚57402.5537109375 disk.rbd0.in‚0 network.recv‚ 1.468626e+06) disk./.usagePercent‚76.50578183123041( disk.nvme0c0n1.out‚49641.69287109375& disk.nvme0n1.out‚49641.69287109375 disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625 cpu‚26.129918789508125 proc.cpu.threads‚17áÀ^yŸ•‚(:” ÏßôÒýËË gpu.0.gpu‚54.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚191.374' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚63.791333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·Åpž–‚(:“ ÞßôÒ±ŽÌË gpu.0.gpu‚90.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚251.903' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚83.96766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k}Êc—‚(:Ø ÞßôÒ²ÖëË) disk./.usagePercent‚76.50578207970442 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚16674.314453125 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚16674.314453125 disk.dm-0.in‚ 4.3359375 network.recv‚ 1.473038e+06$ disk./.usageGB‚1174.555019378662 disk.dm-1.in‚0" disk.dm-1.out‚8607.18798828125 disk.md0.out‚ 142.43359375# disk.dm-0.out‚57404.00048828125$ memory_percent‚95.49026489257812# proc.memory.availableMB‚738.875 network.sent‚ 1.4483402e+07 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625& disk.nvme0c0n1.out‚49643.244140625$ disk.nvme0n1.out‚49643.244140625* proc.memory.percent‚17.352581024169922 proc.cpu.threads‚17 disk.md0.in‚0! proc.memory.rssMB‚ 2843.046875 cpu‚26.131475396621518{Í2˜- _wandbruntime‚5483  _runtime‚5483¸\ k檀 trainer/global_step‚10549 % _timestamp‚1.7844920071288955e+09 _runtime‚5483.703833496 _step‚853 ' lr-AdamW/pg1‚0.00018206421917933718 ' lr-AdamW/pg2‚0.00018206421917933718<†¶¹ÏšÉ trainer/global_step‚10549 % _timestamp‚1.7844920071288955e+09 _runtime‚5483.703833496 _step‚853 ' lr-AdamW/pg1‚0.00018206421917933718 ' lr-AdamW/pg2‚0.00018206421917933718ÕqTË2›- _wandbruntime‚5483  _runtime‚5483¥ß~Ã'œðO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106695356 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82285216 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82285214 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚809702967710274 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚818190620534272 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106695356 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚799584958314480 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10447230358016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚799584975353840 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 188980570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚818190262229504 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10136231869440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10118349319116 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106705499 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚652333961912320 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚809703324672956 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10136231869440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188990715 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10447101382144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 _runtime‚5483.704427537 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚641886731554304 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚641886707129856 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 _step‚854 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83968287 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚818190262229504 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106705499 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚652333808512000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82285216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚808054047878656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27938720 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 YKDeviceStatsMonitor.on_train_batch_st½Ã3art/reserved_bytes.small_pool.allocated‚ 545259520*lrû'õO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83968287 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚818190262229504 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106705499 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚652333808512000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82285216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚808054047878656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27938720 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106695356 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82285216 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82285214 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚809702967710274 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚818190620534272 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106695356 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚799584958314480 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10447230358016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚799584975353840 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 188980570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚818190262229504 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10136231869440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10118349319116 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106705499 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚652333961912320 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚809703324672956 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10136231869440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188990715 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10447101382144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 _runtime‚5483.704427537 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚641886731554304 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚641886707129856 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 _step‚854Ö£Ûßk2ž- _wandbruntime‚5484  _runtime‚5484ˆñò˜'ŸúM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106715388 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106705231 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56034943 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10119277542464 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188997947 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10137502574080 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82292718 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10137502574080 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82292716 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56034941 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10137161780736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚799658401436320 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189008106 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚652393799875584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83975847 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10118937619030 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82292718 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10448191988736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚809777322015990 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚818265742221824 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚641945607886848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27941188 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚808128222129152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚652393648579584 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚799658384396960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10137161780736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚10549 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83976131 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82292716 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚641945583462400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 _step‚855 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚818265383909888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189008106Ûí ' ÿM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10137161780736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚10549 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83976131 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82292716 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚641945583462400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 _step‚855 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚818265383909888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189008106 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106715388 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106705231 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56034943 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10119277542464 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188997947 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10137502574080 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82292718 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10137502574080 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82292716 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56034941 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10137161780736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚799658401436320 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189008106 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚652393799875584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83975847 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10118937619030 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82292718 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10448191988736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚809777322015990 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚818265742221824 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚641945607886848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27941188 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚808128222129152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚652393648579584 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚799658384396960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0×€ã¢2¡- _wandbruntime‚5484  _runtime‚5484;÷M ¢ƒ _runtime‚5484.076853946 _step‚856 1loss/SRqqt2/detection_loss‚0.7871022218354664 %loss/total_loss‚8.880815374654754 epoch‚13 0loss/FRt1/assignment_loss‚0.7945267004026494 2loss/SRqqt1/assignment_loss‚0.5548341150812545 2loss/FBt1/assignment_loss‚0.007053579413811573 1loss/SRqqt1/detection_loss‚0.6028057159696488 /loss/FBt1/detection_loss‚0.6920522606330147 trainer/global_step‚10549 % _timestamp‚1.7844920075021405e+09 2loss/SRqqt2/assignment_loss‚0.7164316748515561 .loss/FRt1/detection_loss‚1.487675143847408 /loss/FBt2/detection_loss‚0.6277876291424036 0loss/FRt2/assignment_loss‚1.6612442310493771 1loss/FBt2/assignment_loss‚0.28186983548999933 /loss/FRt2/detection_loss‚0.6674322669381635¼7d£ˆ %loss/total_loss‚8.880815374654754 epoch‚13 0loss/FRt1/assignment_loss‚0.7945267004026494 2loss/SRqqt1/assignment_loss‚0.5548341150812545 2loss/FBt1/assignment_loss‚0.007053579413811573 1loss/SRqqt1/detection_loss‚0.6028057159696488 /loss/FBt1/detection_loss‚0.6920522606330147 trainer/global_step‚10549 % _timestamp‚1.7844920075021405e+09 2loss/SRqqt2/assignment_loss‚0.7164316748515561 .loss/FRt1/detection_loss‚1.487675143847408 /loss/FBt2/detection_loss‚0.6277876291424036 0loss/FRt2/assignment_loss‚1.6612442310493771 1loss/FBt2/assignment_loss‚0.28186983548999933 /loss/FRt2/detection_loss‚0.6674322669381635 _runtime‚5484.076853946 _step‚856 1loss/SRqqt2/detection_loss‚0.7871022218354664Ø?t¶ ž¤‚(:“ íßôÒÊì‚Ì gpu.0.gpu‚55.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚261.997' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.33233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É™Ì@n¥‚(:Ñ íßôÒõ¬‘Ì) disk./.usagePercent‚76.50573511811619 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0" disk.dm-1.out‚8607.34716796875 network.recv‚ 1.476219e+06$ disk./.usageGB‚1174.554298400879 disk.rbd0.in‚0 disk.nvme0n1.in‚ 4.3359375# disk.dm-0.out‚57406.99951171875$ memory_percent‚96.99323177337646( proc.memory.avaŽ¡cîilableMB‚ 492.62890625* proc.memory.percent‚17.352581024169922 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375% disk.nvme0c0n1.out‚49646.34765625# disk.nvme0n1.out‚49646.34765625! disk.nvme1c1n1.out‚ 16675.3125 disk.md0.in‚0 disk.md0.out‚ 142.87890625! proc.memory.rssMB‚ 2843.046875 network.sent‚ 1.4499162e+07 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.nvme1n1.out‚ 16675.3125 cpu‚26.133339829937384 proc.cpu.threads‚17%Œè_¦‚(:Ô üßôÒªæËË disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚49833.94873046875 disk.nvme1c1n1.in‚0 disk.nvme1n1.out‚ 16735.625 disk.md0.in‚0$ memory_percent‚96.99342250823975 network.recv‚ 1.480521e+06% disk./.usageGB‚1174.7922554016113 disk.dm-1.in‚0( disk.nvme0c0n1.out‚49833.94873046875 disk.md0.out‚ 143.48046875 proc.cpu.threads‚17( disk./.usagePercent‚76.5212346784961 disk.dm-0.in‚ 4.3359375" disk.dm-0.out‚57653.4404296875 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1c1n1.out‚ 16735.625 disk.nvme1n1.in‚0 network.sent‚ 1.4550555e+07" disk.dm-1.out‚8607.54638671875( proc.memory.availableMB‚ 492.59765625! proc.memory.rssMB‚ 2843.046875* proc.memory.percent‚17.352581024169922 cpu‚26.134930680559577ñÌbž§‚(:“ üßôÒ‘LjÌ gpu.0.gpu‚55.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚225.825' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚75.27499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 }¿2¨- _wandbruntime‚5509  _runtime‚5509mžŸoÊ©Ä _runtime‚5509.360010036 _step‚857 ' lr-AdamW/pg1‚0.00018176399239597216 ' lr-AdamW/pg2‚0.00018176399239597216 trainer/global_step‚10599 % _timestamp‚1.7844920327850134e+09…|ö컃 ' lr-AdamW/pg1‚0.00018176399239597216 ' lr-AdamW/pg2‚0.00018176399239597216 trainer/global_step‚10599 % _timestamp‚1.7844920327850134e+09 _runtime‚5509.360010036 _step‚857Ù`×äõ2«- _wandbruntime‚5509  _runtime‚5509 Ú ¹ö'¬ðO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚811763618814464 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107201101 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚811763636333056 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚803256279477840 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚813421071861964 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚655325560065024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚821946378505728 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚811763618814464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚644830523757056 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚811763636333056 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107190958 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚821946378505728 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10182759691264 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10183100477440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 135267328 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82660314 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10495036307968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84348251 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84347976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28064066 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚10599 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82660316 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10183100477440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0%à û'­õO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10183100477440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 135267328 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82660314 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10495036307968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84348251 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84347976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28064066 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚10599 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82660316 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10183100477440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚811763618814464 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107201101 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚811763636333056 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚803256279477840 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚813421071861964 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚655325560065024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚821946378505728 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚811763618814464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚644830523757056 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚811763636333056 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107190958 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚821946378505728 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10182759691264 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145ÚüNO*2®- _wandbruntime‚5509  _runtime‚5509k4c¨&¯úM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28066517 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107210990 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚822021500186112 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10496127430656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚813495069204998 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 _step‚859 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚655385396454400 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 _runtime‚5509.735373094 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82667818 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10183689602560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107200833 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107200833 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82667816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚811837810583552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10183689602560 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189878808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚644889400089600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10165380684038 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 trainer/global_step‚10599 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚811837828102144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10165720607472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚822021858498048 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84355508 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10495996364800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56288993 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82667816 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10184030395904 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107210990 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 \HDeviceStatsMonitor.on_train_batch_end/inaT+!Xctive_split_bytes.all.allocated‚655385551944704 % _timestamp‚1.7844920331602535e+09aä€Ú'°ÿM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚811837810583552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10183689602560 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189878808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚644889400089600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10165380684038 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 trainer/global_step‚10599 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚811837828102144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10165720607472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚822021858498048 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84355508 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10495996364800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56288993 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82667816 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10184030395904 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107210990 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚655385551944704 % _timestamp‚1.7844920331602535e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28066517 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107210990 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚822021500186112 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10496127430656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚813495069204998 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 _step‚859 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚655385396454400 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 _runtime‚5509.735373094 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82667818 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10183689602560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107200833 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107200833 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82667816Ûd>} 2±- _wandbruntime‚5509  _runtime‚5509bÕ•¼ ²„ 0loss/FRt2/assignment_loss‚1.7854165582693824 2loss/SRqqt1/assignment_loss‚0.6160992684606607 2loss/SRqqt2/assignment_loss‚0.7084340101959102 1loss/SRqqt2/detection_loss‚0.7592764046096888 % _timestamp‚1.7844920331612968e+09 /loss/FRt1/assignment_loss‚0.680658516414622 1loss/FBt2/assignment_loss‚0.03151742754727138 /loss/FRt1/detection_loss‚1.5636689296361317 epoch‚13 _runtime‚5509.739417641 _step‚860 0loss/FBt2/detection_loss‚0.44812022856521333 1loss/FBt1/assignment_loss‚0.02869376359922627 /loss/FRt2/detection_loss‚0.5317346164873776 1loss/SRqqt1/detection_loss‚0.6424796907617026 0loss/FBt1/detection_loss‚0.44897223141655995 trainer/global_step‚10599 %loss/total_loss‚8.245071645963748Ÿ&³Ð³‰ 0loss/FRt2/assignment_loss‚1.7854165582693824 2loss/SRqqt1/assignment_loss‚0.6160992684606607 2loss/SRqqt2/assignment_loss‚0.7084340101959102 1loss/SRqqt2/detection_loss‚0.7592764046096888 % _timestamp‚1.7844920331612968e+09 /loss/FRt1/assignment_loss‚0.680658516414622 1loss/FBt2/assignment_loss‚0.03151742754727138 /loss/FRt1/detection_loss‚1.5636689296361317 epoch‚13 _runtime‚5509.739417641 _step‚860 0loss/FBt2/detection_loss‚0.44812022856521333 1loss/FBt1/assignment_loss‚0.02869376359922627 /loss/FRt2/detection_loss‚0.5317346164873776 1loss/SRqqt1/detection_loss‚0.6424796907617026 0loss/FBt1/detection_loss‚0.44897223141655995 trainer/global_step‚10599 %loss/total_loss‚8.245071645963748Üu¤\j´‚(:ß ‹àôÒñëË network.sent‚ 1.4553578e+07 network.recv‚ 1.48394e+06 disk.nvme1c1n1.in‚0# disk.dm-0.out‚57663.23291015625 disk.rbd0.in‚0* proc.memory.percent‚17.352581024169922) disk./.usagePercent‚76.52226609411893' disk.nvme1c1n1.out‚16737.6533203125 disk.md0.in‚0 disk.md0.out‚ 143.66796875' disk.nvme0c0n1.out‚49842.2451171875 disk.nvme0n1.in‚ 4.3359375$ memory_percent‚96.99466228485107( proc.memory.availableMB‚ 492.39453125$ disk./.usageGB‚1174.808090209961 disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375% disk.nvme0n1.out‚49842.2451171875! proc.memory.rssMB‚ 2843.046875 cpu‚26.13664059967951 proc.cpu.threads‚17 disk.nvme1n1.in‚0% disk.nvme1n1.out‚16737.6533203125 disk.dm-0.in‚ 4.3359375" disk.dm-1.out‚8607.65673828125ð‡M5žµ‚(:“ ‹àôÒ‹ëÏÌ gpu.0.gpu‚74.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚244.353' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚81.45100000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á0ü^¶‚(:Ó šàôÒù“ÎË disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚16741.080078125 disk.nvme1n1.in‚0$ memory_percent‚96.99559211730957 disk.dm-1.in‚0$ disk.nvme0n1.out‚49847.251953125# disk.dm-0.out‚57667.99658203125* proc.memory.percent‚17.352581024169922 cpu‚26.138300916740597 proc.cpu.threads‚17" disk.dm-1.out‚8610.60595703125 disk.nvme0c0n1.in‚ 4.3359375$ disk.nvme1n1.out‚16741.080078125 disk.md0.in‚0 disk.md0.out‚144 disk.dm-0.in‚ 4.3359375' proc.memory.availableMB‚ 492.2421875! proc.memory.rssMB‚ 2843.046875) disk./.usagePercent‚76.52228025713761 disk.rbd0.in‚0& disk.nvme0c0n1.out‚49847.251953125 disk.nvme0n1.in‚ 4.3359375 network.sent‚ 1.4616996e+07 network.recv‚ 1.488263e+06$ disk./.usageGB‚1174.808307647705 disk.rbd0.out‚ 0.4697265625,•ñw‘·‚(:† šàôÒ냕Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚52.0 gpu.0.powerWatts‚117.915' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.305 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰Üq­2¸- _wandbruntime‚5535  _runtime‚5535åjòȹ trainer/global_step‚10649 % _timestamp‚1.7844920584809413e+09 _runtime‚5535.055893955 _step‚861 & lr-AdamW/pg1‚0.0001814625588997894 & lr-AdamW/pg2‚0.0001814625588997894k7®ÍºÇ _runtime‚5535.055893955 _step‚861 & lr-AdamW/pg1‚0.0001814625588997894 & lr-AdamW/pg2‚0.0001814625588997894 trainer/global_step‚10649 % _timestamp‚1.7844920584809413e+09Ýý«èç2»- _wandbruntime‚5535  _runtime‚5535Hˆ5ö'¼ðO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83035416 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚647774364808704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 _step‚862 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10229628299264 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚647774340384256 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚815473224787456 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10542977595904 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83035416 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚817138462088290 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84727767 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10543110766080 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 _runtime‚5535.056809436 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚825702853086720 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28190089 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚658317317980160 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚815473224787456 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28189807 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56537960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚815473207268864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107686560 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 % _timestamp‚1.7844920584818754e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10229287513088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚815473207268864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107696703 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84728051 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107686560 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚825702494781952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚806927583601840 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10211235449132 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83035414 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚817138819050972 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10210895525810 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 trainer/global_step‚10649 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚806927566562480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10229287513088 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 190721974 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10229628299264 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 190732119 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83035414 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚825702853086720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56537962 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚825702494781952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83035416 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚817138462088290 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84727767 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10543110766080 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 _runtime‚5535.056809436 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚825702853086720 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28190089 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚658317317980160 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚815473224787456 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28189807 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56537960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚815473207268864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107686560 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 % _timestamp‚1.7844920584818754e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10229287513088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚815473207268864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107696703 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84728051 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107686560 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚825702494781952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚806927583601840 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10211235449132 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83035414 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚817138819050972 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10210895525810 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 trainer/global_step‚10649 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚806927566562480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10229287513088 WGDeviceSta¡¦ò tsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 190721974 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10229628299264 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 190732119 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83035414 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚825702853086720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56537962 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚825702494781952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 190739351 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚815547416556544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56543041 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 _step‚863 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10230217424384 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84735356 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚815547416556544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28192594 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10544063430656 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚825777974774272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83042918 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190749510 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚658377149081600 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚825777616462336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 190739351 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚807000992644960 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚10649 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83042918 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107706592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 _runtime‚5535.430449082 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107706592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚817213173356800 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10230558217728 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚647833216716800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚825777974774272 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚807001009684320 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 Œ¸'ÀÿM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚815547399037952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107696435 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28192315 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10212163672480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56543043 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10543932364800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10230217424384 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10230558217728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚825777974774272 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚807001009684320 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 190739351 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚815547416556544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56543041 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 _step‚863 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10230217424384 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84735356 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚815547416556544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28192594 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10544063430656 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚825777974774272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83042918 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190749510 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚658377149081600 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚825777616462336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 190739351 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚807000992644960 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚10649 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83042918 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107706592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 _runtime‚5535.430449082 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107706592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚817213173356800 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10230558217728 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚647833216716800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚829458969362944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚829458611058176 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10275815334912 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚298 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚829458611058176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56792012 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85108216 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83410514 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚819182795723264 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚820856566239980 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚820856209277298 trainer/global_step‚10699 _runtime‚5559.657334389 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108192305 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚819182813241856 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85107918 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108192305 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚650718181435904 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322ÕÐû'ËõO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83410514 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚819182795723264 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚820856566239980 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚820856209277298 trainer/global_step‚10699 _runtime‚5559.657334389 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108192305 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚819182813241856 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85107918 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108192305 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚650718181435904 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚650718157011456 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚819182813241856 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28316204 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚829458969362944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚829458611058176 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10275815334912 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚298 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚829458611058176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56792012 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85108216 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0â$ë+2Ì- _wandbruntime‚5560  _runtime‚5560Ñn¹ 'ÍúM OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108192037 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚661369127075328 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚829533732738560 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191620212 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚661368977876480 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚829534091050496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85115461 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28318642 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 _runtime‚5560.331755549 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚820930563583014 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚819257005010944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10258266814054 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7844920837566195e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10277086039552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10258606737488 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191610053 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85115735 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83418018 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10276745246208 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚829534091050496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83418018 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56797093 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28318370 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108192037 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚650777057768448 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚810672296768960 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚819256987492352 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83418016 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191610053 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10277086039552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚819257005010944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157@½qÒâ"ÎÿM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108202194 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10276745246208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83418016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚829534091050496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85115461 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28318642 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 _runtime‚5560.331755549 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚820930563583014 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚819257005010944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10258266814054 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7844920837566195e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10277086039552 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10258606737488 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191610053 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85115735 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83418018 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10276745246208 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚829534091050496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83418018 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56797093 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28318370 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108192037 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚650777057768448 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚810672296768960 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚819256987492352 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83418016 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191610053 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10277086039552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚819257005010944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108192037 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚661369127075328 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚829533732738560 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191620212 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚661368977876480 UGDeviceStatsMonitor.on_train_batch_end/ûÁn¬#reserved_bytes.small_pool.current‚ 545259520 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚833214727334400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚833215085639168 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108677764 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚653661998063104 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10304121579148 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83785614 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚664301107865088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57046060 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚822892401696256 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 trainer/global_step‚10749 _step‚870 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28442529 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 % _timestamp‚1.7844921071059155e+09 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10322683942912 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚822892384177664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10303781655826 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108677764 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 192473523 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83785614 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10322683942912 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚824574313428988 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108687907 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚833215085639168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚824573956466306 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57046062 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚822892384177664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85488308 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10639109801984 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 _runtime‚ 5583.68441142 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚814270174810480 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83785616 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10322343156736 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚653661973638656 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85488591 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10638980826112ý.r…ú'ÛôO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83785614 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚664301107865088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57046060 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚822892401696256 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 trainer/global_step‚10749 _step‚870 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28442529 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 % _timestamp‚1.7844921071059155e+09 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10322683942912 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚822892384177664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10303781655826 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108677764 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 192473523 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83785614 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10322683942912 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚824574313428988 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108687907 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚833215085639168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚824573956466306 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57046062 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚822892384177664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85488308 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10639109801984 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 _runtime‚ 5583.68441142 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚814270174810480 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83785616 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10322343156736 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚653661973638656 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85488591 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10638980826112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108687907 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚822892401696256 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚833214727334400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚833215085639168 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108677764 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚653661998063104 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10304121579148 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784æö˜·2Ü- _wandbruntime‚5584  _runtime‚5584^Uç§ôÝúM YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚814343600892960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83793118 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28444937 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10323613861376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚5584.040828899 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10323273068032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 192480755 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57051141 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚824648310772022 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108697796 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 192490914 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57051143 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚833289849014784 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚822966575946752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚664360942903808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚653720874395648 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚814343617932320 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚824648667734816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7844921074656963e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 192490914 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚871 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10323273068032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚822966593465344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚833290207326720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚822966593465344 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚653720849971200 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 192480755 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108687639 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83793118 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10304709879062 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peB‡í eak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83793116 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10323613861376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85495786 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108697796 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108687639 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83793118 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28444937 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10323613861376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚5584.040828899 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10323273068032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 192480755 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57051141 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚824648310772022 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108697796 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 192490914 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57051143 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚833289849014784 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚822966575946752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚664360942903808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚653720874395648 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚814343617932320 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚824648667734816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7844921074656963e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 192490914 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚871 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10323273068032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚822966593465344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚833290207326720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚822966593465344 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚653720849971200 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 192480755 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108687639 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83793118 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10304709879062 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83793116 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10323613861376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85495786 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108697796 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108687639 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520çpý2ß- _wandbruntime‚5584  _runtime‚5584$ ‚ à† epoch‚13 trainer/global_step‚10749 0loss/FRt1/assignment_loss‚0.8063010043568082 0loss/FRt2/assignment_loss‚1.7144901752471924 2loss/FBt2/assignment_loss‚0.011980415485290427 1loss/SRqqt1/detection_loss‚0.5942741724058266 %loss/total_loss‚8.060216866062452 _step‚872 2loss/SRqqt2/assignment_loss‚0.6914895466343347 3loss/FBt1/assignment_loss‚0.0017764580132215713 /loss/FRt1/detection_loss‚1.4200020913526208 _runtime‚ 5584.04133742 1loss/SRqqt2/detection_loss‚0.7942534911653234 /loss/FBt1/detection_loss‚0.5970117729363753 0loss/FBt2/detection_loss‚0.37205843201005145 % _timestamp‚1.7844921074666476e+09 2loss/SRqqt1/assignment_loss‚0.5306662412400619 /loss/FRt2/detection_loss‚0.5259130652153459^-xÿá‹ /loss/FRt2/detection_loss‚0.5259130652153459 epoch‚13 trainer/global_step‚10749 0loss/FRt1/assignment_loss‚0.8063010043568082 0loss/FRt2/assignment_loss‚1.7144901752471924 2loss/FBt2/assignment_loss‚0.011980415485290427 1loss/SRqqt1/detection_loss‚0.5942741724058266 %loss/total_loss‚8.060216866062452 _step‚872 2loss/SRqqt2/assignment_loss‚0.6914895466343347 3loss/FBt1/assignment_loss‚0.0017764580132215713 /loss/FRt1/detection_loss‚1.4200020913526208 _runtime‚ 5584.04133742 1loss/SRqqt2/detection_loss‚0.7942534911653234 /loss/FBt1/detection_loss‚0.5970117729363753 0loss/FBt2/detection_loss‚0.37205843201005145 % _timestamp‚1.7844921074666476e+09 2loss/SRqqt1/assignment_loss‚0.5306662412400619èåþ¬lâ‚(:á ÖàôÒý¬ÜË% disk./.usageGB‚1172.6430892944336 disk.nvme0c0n1.in‚ 4.3359375' disk.nvme0c0n1.out‚50094.3662109375 disk.nvme0n1.in‚ 4.3359375& disk.nvme1n1.out‚17677.64501953125 disk.dm-1.in‚0! disk.dm-1.out‚8611.9755859375 network.recv‚ 1.503616e+06% disk.nvme0n1.out‚50094.3662109375( disk.nvme1c1n1.out‚17677.64501953125 disk.dm-0.in‚ 4.3359375 disk.rbd0.in‚0$ memory_percent‚96.99194431304932( proc.memory.availableMB‚ 492.83984375 proc.cpu.threads‚17 network.sent‚ 1.4751493e+07 disk.md0.in‚0 disk.rbd0.out‚ 0.4697265625! proc.memory.rssMB‚ 2843.046875* proc.memory.percent‚17.352581024169922 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 145.6328125# disk.dm-0.out‚58846.85302734375 cpu‚26.13561696052545) disk./.usagePercent‚76.38124665653369–hJOœã‚(:‘ ÖàôÒœÆÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚221.848' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.94933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ëu{„2ä- _wandbruntime‚5607  _runtime‚5607f%Ü«ÊåÄ ' lr-AdamW/pg2‚0.00018055106651120294 trainer/global_step‚10799 % _timestamp‚1.7844921311289015e+09 _runtime‚5607.703882438 _step‚873 ' lr-AdamW/pg1‚0.00018055106651120294zý12ÏæÉ _runtime‚5607.703882438 _step‚873 ' lr-AdamW/pg1‚0.00018055106651120294 ' lr-AdamW/pg2‚0.00018055106651120294 trainer/global_step‚10799 % _timestamp‚1.7844921311289015e+09é¿âÎ2ç- _wandbruntime‚5607  _runtime‚56079Á¤¸õ'èïO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 trainer/global_step‚10799 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚836971201915392 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.784492131129833e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚836971201915392 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84160716 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚817941478934480 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚667292724198912 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚826601972632064 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109173366 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85868106 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85867826 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10368870978560 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193334080 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193344225 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84160714 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28567716 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84160716 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57300110 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10369211764736 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚826601972632064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚836970843610624 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚828292060617996 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84160714 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚826601990150656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109173366 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚836970843610624 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚836971201915392 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.784492131129833e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚836971201915392 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84160716 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚817941478934480 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚667292724198912 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚826601972632064 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109173366 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85868106 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85867826 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10368870978560 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193334080 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193344225 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84160714 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28567716 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84160716 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57300110 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10369211764736 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚826601972632064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚836970843610624 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚828292060617996 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.fre1/bq¯ed‚84160714 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚826601990150656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109173366 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚836970843610624 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193351457 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚656664691022848 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57305191 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84168218 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84168218 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109193398 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚826676181919744 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚837046323602944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚826676164401152 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193361616 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚826676164401152 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109183241 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚667352707587584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 % _timestamp‚1.7844921314886572e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84168216 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28570537 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚667352556291584 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10688016564736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84168216 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109193398 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193361616 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚818014922056320 trainer/global_step‚10799 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85875452 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10687889693184 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10351492867504 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚826676181919744 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28570261 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚875 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109183241 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10351152944070 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚837045965291008 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193351457 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10370141683200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 _runtime‚5608.063891256 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10370141683200 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚656664666598400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚828366414923824 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚818014905016960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85875452 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10687889693184 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10351492867504 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚826676181919744 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28570261 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚875 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109183241 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10351152944070 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚837045965291008 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193351457 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10370141683200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 _runtime‚5608.063891256 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10370141683200 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚656664666598400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚828366414923824 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚818014905016960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193351457 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚656664691022848 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57305191 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84168218 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84168218 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109193398 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚826676181919744 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚837046323602944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚826676164401152 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193361616 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚826676164401152 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109183241 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚667352707587584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 % _timestamp‚1.7844921314886572e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84168216 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28570537 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚667352556291584 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10688016564736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84168216 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675ëvZäF2í- _wandbruntime‚5608  _runtime‚5608úI( î„ /loss/FBt2/detection_loss‚0.4519387446343899 epoch‚13 trainer/global_step‚10799 1loss/FBt1/assignment_loss‚0.01816084591773233 /loss/FRt2/detection_loss‚0.5631528097669126 0loss/FBt1/detection_loss‚0.34759985328962406 _runtime‚5608.064396017 1loss/FBt2/assignment_loss‚0.02916789545090483 %loss/total_loss‚7.447331786011024 _step‚876 /loss/FRt2/assignment_loss‚1.683798881333608 1loss/SRqqt1/detection_loss‚0.5716061324683122 1loss/SRqqt2/detection_loss‚0.7953658910478685 % _timestamp‚1.7844921314897091e+09 0loss/FRt1/assignment_loss‚0.7171738976572716 3loss/SRqqt1/assignment_loss‚0.44819289692815495 2loss/SRqqt2/assignment_loss‚0.5142982152412944 /loss/FRt1/detection_loss‚1.3068757222749516,èúÑï‰ /loss/FRt2/assignment_loss‚1.683798881333608 1loss/SRqqt1/detection_loss‚0.5716061324683122 1loss/SRqqt2/detection_loss‚0.7953658910478685 % _timestamp‚1.7844921314897091e+09 0loss/FRt1/assignment_loss‚0.7171738976572716 3loss/SRqqt1/assignment_loss‚0.44819289692815495 2loss/SRqqt2/assignment_loss‚0.5142982152412944 /loss/FRt1/detection_loss‚1.3068757222749516 /loss/FBt2/detection_loss‚0.4519387446343899 epoch‚13 trainer/global_step‚10799 1loss/FBt1/assignment_loss‚0.01816084591773233 /loss/FRt2/detection_loss‚0.5631528097669126 0loss/FBt1/detection_loss‚0.34759985328962406 _runtime‚5608.064396017 1loss/FBt2/assignment_loss‚0.02916789545090483 %loss/total_loss‚7.447331786011024 _step‚876ì–ÙÑ[mð‚(:â åàôÒÈךÌ) disk./.usagePercent‚76.38125585007214 disk.dm-1.out‚8612.091796875 disk.rbd0.in‚0( disk.nvme0c0n1.out‚50104.48486328125 disk.nvme0n1.in‚ 4.3359375( disk.nvme1c1n1.out‚17680.17626953125 disk.nvme1n1.in‚0( proc.memory.availableMB‚ 493.04296875 network.sent‚ 1.4813893e+07 network.recv‚ 1.507978e+06& disk.nvme0n1.out‚50104.48486328125 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚17680.17626953125 disk.md0.in‚0! disk.dm-0.out‚58858.416015625! proc.memory.rssMB‚ 2843.046875 disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625 disk.md0.out‚ 146.08203125 disk.dm-0.in‚ 4.3359375$ memory_percent‚96.99070453643799% disk./.usageGB‚1172.6432304382324 disk.nvme0c0n1.in‚ 4.3359375* proc.memory.percent‚17.352581024169922 cpu‚26.134583977566525 proc.cpu.threads‚17^×€ýžñ‚(:“ åàôÒâÇðÌ gpu.0.gpu‚91.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚245.765' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚81.92166666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„tumò‚(:â ôàôÒ©“ÑË% disk./.usageGB‚1172.6431045532227 disk.dm-0.in‚ 4.3359375! disk.dm-0.out‚58865.607421875 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 4.3359375& disk.nvme0n1.out‚50108.78076171875& disk.nvme1n1.out‚17683.65185546875 disk.md0.in‚0 network.sent‚ 1.4875136e+07 network.recv‚ 1.512421e+06) disk./.usagePercent‚76.38124765042974 disk.dm-1.out‚8612.173828125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚50108.78076171875 disk.nvme0n1.in‚ 4.3359375( disk.nvme1c1n1.out‚17683.65185546875 disk.rbd0.out‚ 0.4697265625 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0( proc.memory.availableMB‚ 492.55859375* proc.memory.percent‚17.352581024169922 disk.md0.out‚ 146.30859375$ memory_percent‚96.99366092681885! proc.memory.rssMB‚ 2843.046875 cpu‚26.133654893004074 proc.cpu.threads‚17FG¿ó‚(:’ ôàôÒÏÚ–Ì gpu.0.gpu‚91.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚240.98' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚80.32666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v@·!2ô- _wandbruntime‚5631  _runtime‚5631˜<ö·ÊõÄ ' lr-AdamW/pg1‚0.00018024485463655974 ' lr-AdamW/pg2‚0.00018024485463655974 trainer/global_step‚10849 % _timestamp‚1.7844921551383696e+09 _runtime‚5631.713365887 _step‚877Pˆ_ÄÏöÉ ' lr-AdamW/pg1‚0.00018024485463655974 ' lr-AdamW/pg2‚0.00018024485463655974 trainer/global_step‚10849 % _timestamp‚1.7844921551383696e+09 _runtime‚5631.713365887 _step‚877í7.k2÷- _wandbruntime‚5631  _runtime‚5631!FóM øðO [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚840726959886848 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84535816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚840726959886848 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚821612783058480 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84535816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚878 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57554160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚830311578605056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84535814 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freཱི¹©ed‚10734983608832 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 _runtime‚5631.717367973 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚840727318191616 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86248733 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 194204782 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84535814 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109668968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚840727318191616 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚821612800097840 e_îû'ùõO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10415739586560 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57554162 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28694859 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86249021 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚659549606893056 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10415398800384 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10415398800384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚830311561086464 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10397007709164 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109679111 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109679111 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10396667785842 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109668968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚832009450844322 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10415739586560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28694573 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 _runtime‚5631.717367973 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚840727318191616 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86248733 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 194204782 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84535814 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109668968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚840727318191616 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚821612800097840 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚840726959886848 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84535816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚840726959886848 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚821612783058480 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84535816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚878 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57554160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚830311578605056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84535814 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10734983608832 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159684608 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10397935932512 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10416669505024 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109689000 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109678843 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚840802081567232 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57559243 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚821686209140960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109678843 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86256578 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10735939536896 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚821686226180320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚10849 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109689000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚670344582447104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86256286 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135260160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84543316 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 194222159 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194232318 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚5632.071211412 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84543318 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚840802439879168 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚830385770374144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10397596009078X’ÉüþM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10416669505024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚659608483225600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚830385752855552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28697335 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10736074797056 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 194222159 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194232318 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚5632.071211412 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84543318 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚840802439879168 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚830385770374144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10397596009078 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84543316 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57559241 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159684608 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10397935932512 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10416669505024 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109689000 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109678843 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚840802081567232 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57559243 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚821686209140960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109678843 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86256578 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10735939536896 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚821686226180320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚10849 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109689000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚670344582447104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86256286 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135260160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84543316 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B“ƒ‚(:ˆ ’áôÒ…€§Ì gpu.0.gpu‚88.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚256.248' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚85.416 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[8<ˆ2„- _wandbruntime‚5655  _runtime‚5655—)uÇ…Á & lr-AdamW/pg1‚0.0001799374603811735 & lr-AdamW/pg2‚0.0001799374603811735 trainer/global_step‚10899 % _timestamp‚1.7844921791395922e+09 _runtime‚ 5655.71458997 _step‚881¢^ŠÄÌ†Æ & lr-AdamW/pg1‚0.0001799374603811735 & lr-AdamW/pg2‚0.0001799374603811735 trainer/global_step‚10899 % _timestamp‚1.7844921791395922e+09 _runtime‚ 5655.71458997 _step‚881ñ/O¥Â2‡- _wandbruntime‚5655  _runtime‚5655d^&Üö'ˆðO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110174713 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10461926622208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110174713 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84910916 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84910914 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚844483434467840 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚834021167059456 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10462267408384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110164570 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10782929149440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚10899 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 % _timestamp‚1.7844921791405683e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚673276352669696 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚844483076163072 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10462267408384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚834021167059456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10461926622208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86628990 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚673276503972864 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚844483434467840 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚834021149540864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 _step‚882 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57808210 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28820498 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚835727554996012 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚825284087182480 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84910916 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10783056028160 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚825284104221840 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110164570 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚662493423520256 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10443110850850 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57808212 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚662493447944704Q8ûû'‰õO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚825284087182480 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84910916 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10783056028160 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚825284104221840 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110164570 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚662493423520256 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10443110850850 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57808212 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚662493447944704 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110174713 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10461926622208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110174713 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84910916 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84910914 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚844483434467840 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚834021167059456 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10462267408384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110164570 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10782929149440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚10899 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 % _timestamp‚1.7844921791405683e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚673276352669696 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚844483076163072 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10462267408384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚834021167059456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10461926622208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86628990 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚673276503972864 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚844483434467840 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚834021149540864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 _step‚882 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57808210 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28820498 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚835727554996012 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935òݧIŠ2Š- _wandbruntime‚5656  _runtime‚5656þà ¼+ ‹úM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚662552324277248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚662552299852800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86636299 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10462856533504 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10444378997520 % _timestamp‚1.7844921794985533e+09 _runtime‚5656.073710399 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚673336189874176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195103020 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28823285 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10463197326848 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚844558197843456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10462856533504 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84918418 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57813293 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110184602 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚673336341170176 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚835801552339046 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚834095358828544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 trainer/global_step‚10899 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚883 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚834095341309952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57813291 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84918416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360k9'ŒÿM F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10462856533504 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84918418 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57813293 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110184602 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚673336341170176 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚835801552339046 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚834095358828544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 trainer/global_step‚10899 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚883 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚834095341309952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57813291 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84918416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚662552324277248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚662552299852800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86636299 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10462856533504 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10444378997520 % _timestamp‚1.7844921794985533e+09 _runtime‚5656.073710399 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚673336189874176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195103020 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28823285 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10463197326848 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚844558197843456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928ó&f#2- _wandbruntime‚5656  _runtime‚5656𯹳 Ž† /loss/FBt2/detection_loss‚0.5994937772552172 1loss/FBt1/assignment_loss‚0.00502502127653823 1loss/SRqqt1/detection_loss‚0.6746900494085666 /loss/FBt1/detection_loss‚0.5451929359818434 epoch‚13 0loss/FRt1/assignment_loss‚0.7410631705092513 0loss/FRt2/assignment_loss‚1.6620421661170264 2loss/SRqqt2/assignment_loss‚0.6266745014271967 3loss/FBt2/assignment_loss‚0.0006309530896184394 1loss/SRqqt2/detection_loss‚0.7290662931676825 %loss/total_loss‚8.011261367392146 % _timestamp‚1.7844921794995358e+09 _step‚884 /loss/FRt1/detection_loss‚1.4494449357081154 /loss/FRt2/detection_loss‚0.5395375416450227 trainer/global_step‚10899 _runtime‚5656.074200839 3loss/SRqqt1/assignment_loss‚0.43840002180606624’šâî‹ 1loss/FBt1/assignment_loss‚0.00502502127653823 1loss/SRqqt1/detection_loss‚0.6746900494085666 /loss/FBt1/detection_loss‚0.5451929359818434 epoch‚13 0loss/FRt1/assignment_loss‚0.7410631705092513 0loss/FRt2/assignment_loss‚1.6620421661170264 2loss/SRqqt2/assignment_loss‚0.6266745014271967 3loss/FBt2/assignment_loss‚0.0006309530896184394 1loss/SRqqt2/detection_loss‚0.7290662931676825 %loss/total_loss‚8.011261367392146 % _timestamp‚1.7844921794995358e+09 _step‚884 /loss/FRt1/detection_loss‚1.4494449357081154 /loss/FRt2/detection_loss‚0.5395375416450227 trainer/global_step‚10899 _runtime‚5656.074200839 3loss/SRqqt1/assignment_loss‚0.43840002180606624 /loss/FBt2/detection_loss‚0.5994937772552172ôYøã i‚(:Þ ¡áôÒŽÍíË network.recv‚ 1.522356e+06 disk.dm-1.in‚0' proc.memory.availableMB‚ 492.7109375* proc.memory.percent‚17.352581024169922 cpu‚26.130703290808885 proc.cpu.threads‚17 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚19599.064453125 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚19599.064453125 disk.md0.in‚0 disk.md0.out‚ 147.3046875 disk.dm-0.in‚ 4.3359375 network.sent‚ 1.4946254e+07) disk./.usagePercent‚76.54566265554075% disk./.usageGB‚1175.1672859191895 disk.rbd0.in‚0( disk.nvme0c0n1.out‚50118.08544921875" disk.dm-1.out‚8612.56298828125$ memory_percent‚96.99273109436035! proc.memory.rssMB‚ 2843.046875 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375& disk.nvme0n1.out‚50118.08544921875! disk.dm-0.out‚60787.779296875}£s‘‚(:’ ¡áôÒ¡¦ÐÌ gpu.0.gpu‚17.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚118.657' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.55233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñÉ2’- _wandbruntime‚5679  _runtime‚5679 잊Ê“Ä ' lr-AdamW/pg1‚0.00017962888866858716 ' lr-AdamW/pg2‚0.00017962888866858716 trainer/global_step‚10949 % _timestamp‚1.7844922027808115e+09 _runtime‚5679.355785876 _step‚885ø~½Ï”É ' lr-AdamW/pg1‚0.00017962888866858716 ' lr-AdamW/pg2‚0.00017962888866858716 trainer/global_step‚10949 % _timestamp‚1.7844922027808115e+09 _runtime‚5679.355785876 _step‚885õ Ù©“2•- _wandbruntime‚5679  _runtime‚5679ëý~ö'–ðO VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 _step‚886 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10508795230208 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10508795230208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85286016 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87008315 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚828955408345840 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10831006121984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 _runtime‚5679.359362772 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚837730755513856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28946329 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10489553915858 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚848239550744064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195956331 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110670315 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚837730755513856 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚676268270693888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85286014 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚837730737995264 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10508454444032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58062262 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10508454444032 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110660172 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87008591 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚848239192439296 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 _step‚886 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10508795230208 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10508795230208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85286016 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87008315 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚828955408345840 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10831006121984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 _runtime‚5679.359362772 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚837730755513856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28946329 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10489553915858 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚848239550744064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195956331 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110670315 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚837730755513856 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚676268270693888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85286014 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚837730737995264 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10508454444032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58062262 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10508454444032 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110660172 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87008591 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚848239192439296 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110680204 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87015917 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28948576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85293518 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87016184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10490822062528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚837804929764352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85293518 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 _step‚887 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195973722 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85293516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28948841 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10490482139094 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚837804929764352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚829028834428320 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚839519656490848 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚848314672431616 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚676328096999936 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚839519299528054 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195973722 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 % _timestamp‚1.7844922031392052e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 195963563 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110670047 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 145004544 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚837804947282944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110680204 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58067343 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚848314314119680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85293516 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚665496116480000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚5679.714362801 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195963563 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚676327951995392 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110670047 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10831835515392 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10509725148672 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10509384355328 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10509725148672 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚665496140904448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10509384355328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚848314672431616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚829028817388960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚837804947282944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 trainer/global_step‚10949T|ß-'šÿM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚837804947282944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110680204 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58067343 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚848314314119680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85293516 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚665496116480000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚5679.714362801 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195963563 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚676327951995392 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110670047 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10831835515392 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10509725148672 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10509384355328 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10509725148672 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚665496140904448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10509384355328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚848314672431616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚829028817388960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚837804947282944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 trainer/global_step‚10949 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110680204 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87015917 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28948576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85293518 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87016184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10490822062528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚837804929764352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85293518 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 _step‚887 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195973722 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85293516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28948841 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10490482139094 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚837804929764352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚829028834428320 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚839519656490848 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚848314672431616 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚676328096999936 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚839519299528054 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195973722 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 % _timestamp‚1.7844922031392052e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 195963563 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110670047 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 145004544 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0÷üZ|§2›- _wandbruntime‚5679  _runtime‚5679Üs@œ‚ 1loss/SRqqt1/detection_loss‚0.7305760067562326 % _timestamp‚1.7844922031401846e+09 1loss/FBt1/assignment_loss‚0.03703563373204565 /loss/FRt1/detection_loss‚1.4258943818562295 0loss/SRqqt2/detection_loss‚0.869407697864757 /loss/FBt1/detection_loss‚0.4832842980201046 _step‚888 /loss/FBt2/detection_loss‚0.5600733754690737 _runtime‚5679.714955562 1loss/FBt2/assignment_loss‚0.04035747094312603 %loss/total_loss‚8.361677954505836 epoch‚13 trainer/global_step‚10949 0loss/FRt1/assignment_loss‚0.6609176866850763 0loss/FRt2/assignment_loss‚1.7482669631871142 2loss/SRqqt1/assignment_loss‚0.6126543764834959 2loss/SRqqt2/assignment_loss‚0.5915629634935132 /loss/FRt2/detection_loss‚0.6016471000150696+§œ ‡ 1loss/FBt2/assignment_loss‚0.04035747094312603 %loss/total_loss‚8.361677954505836 epoch‚13 trainer/global_step‚10949 0loss/FRt1/assignment_loss‚0.6609176866850763 0loss/FRt2/assignment_loss‚1.7482669631871142 2loss/SRqqt1/assignment_loss‚0.6126543764834959 2loss/SRqqt2/assignment_loss‚0.5915629634935132 /loss/FRt2/detection_loss‚0.6016471000150696 1loss/SRqqt1/detection_loss‚0.7305760067562326 % _timestamp‚1.7844922031401846e+09 1loss/FBt1/assignment_loss‚0.03703563373204565 /loss/FRt1/detection_loss‚1.4258943818562295 0loss/SRqqt2/detection_loss‚0.869407697864757 /loss/FBt1/detection_loss‚0.4832842980201046 _step‚888 /loss/FBt2/detection_loss‚0.5600733754690737 _runtime‚5679.714955562øBhjRlž‚(:á °áô҆ؖÌ) disk./.usagePercent‚76.54651343055704$ disk./.usageGB‚1175.180347442627 disk.dm-0.in‚ 4.3359375( disk.nvme0c0n1.out‚50818.33447265625! proc.memory.rssMB‚ 2843.046875* proc.memory.percent‚17.352581024169922 proc.cpu.threads‚17' disk.nvme1c1n1.out‚19827.9345703125 disk.nvme1n1.in‚0! disk.dm-0.out‚61715.861328125 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚50818.33447265625$ memory_percent‚96.99299335479736( proc.memory.availableMB‚ 492.66796875 network.recv‚ 1.526786e+06 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚19827.9345703125 disk.dm-1.in‚0" disk.dm-1.out‚8612.71923828125 disk.rbd0.in‚0 cpu‚26.129785883666827 network.sent‚ 1.5022569e+07 disk.md0.in‚0 disk.md0.out‚ 147.71484375 disk.rbd0.out‚ 0.4697265625ÞVúžŸ‚(:“ °áôÒ…çÎÌ gpu.0.gpu‚66.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚254.049' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.68299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[B¢b_ ‚(:Ô ¿áôÒáÁÕË disk.nvme1n1.in‚0 disk.dm-0.in‚ 4.3359375& proc.memory.availableMB‚ 492.671875 cpu‚26.12888128834011 disk.md0.out‚ 147.96484375 disk.dm-1.in‚0$ memory_percent‚96.99296951293945! proc.memory.rssMB‚ 2843.046875* proc.memory.percent‚17.352581024169922 proc.cpu.threads‚17( disk./.usagePercent‚76.5465062248107% disk./.usageGB‚1175.1802368164062% disk.nvme1n1.out‚19830.1787109375 disk.md0.in‚0# disk.dm-0.out‚61730.34619140625" disk.dm-1.out‚8612.81396484375 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625 network.recv‚ 1.530696e+06 disk.nvme1c1n1.in‚0 disk.nvme0c0n1.in‚ 4.3359375" disk.nvme0c0n1.out‚ 50831.21875 network.sent‚ 1.5070926e+07 disk.nvme0n1.in‚ 4.3359375 disk.nvme0n1.out‚ 50831.21875' disk.nvme1c1n1.out‚19830.1787109375Úý—(ž¡‚(:“ ¿áôÒ‡°Ì gpu.0.gpu‚95.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚269.542' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.84733333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-þ|2¢- _wandbruntime‚5703  _runtime‚5703¦Ùuv£Ä % _timestamp‚1.7844922267217443e+09 _runtime‚5703.297197379 _step‚889 ' lr-AdamW/pg1‚0.00017931©½5ˆT914444120297 ' lr-AdamW/pg2‚0.00017931914444120297 trainer/global_step‚10999b5R Ï¤É _runtime‚5703.297197379 _step‚889 ' lr-AdamW/pg1‚0.00017931914444120297 ' lr-AdamW/pg2‚0.00017931914444120297 trainer/global_step‚10999 % _timestamp‚1.7844922267217443e+09ù„ì”f2¥- _wandbruntime‚5703  _runtime‚5703pFËpõ'¦ïO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10554982265856 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚679259884952576 _runtime‚ 5703.29748897 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚668381056774656 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10554982265856 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 196827033 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 % _timestamp‚1.7844922267224066e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚851995308715520 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚679260044644352 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚832626695430480 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚851995667020288 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10555323052032 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111165917 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87388654 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87388361 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29072342 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 196816888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29072051 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚843162692411346 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85661116 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111165917 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111155774 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10878963445248 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10535996980866 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111155774 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10878828177920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58316310 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10555323052032 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚832626712469840 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚851995308715520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚841440343968256 trainer/global_step‚10999 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚851995667020288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚841440326449664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚890 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85661114 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58316312 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10536336904188 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚843163049374028 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2õ„Sú'§ôO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚851995667020288 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10555323052032 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111165917 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87388654 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87388361 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29072342 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 196816888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29072051 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚843162692411346 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85661116 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111165917 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111155774 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10878963445248 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10535996980866 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111155774 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10878828177920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58316310 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10555323052032 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚832626712469840 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚851995308715520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚841440343968256 trainer/global_step‚10999 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚851995667020288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚841440326449664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚890 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85661114 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58316312 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10536336904188 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚843163049374028 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10554982265856 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚679259884952576 _runtime‚ 5703.29748897 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚668381056774656 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10554982265856 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 196827033 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 % _timestamp‚1.7844922267224066e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚851995308715520 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚679260044644352 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚832626695430480 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0úØ2¨- _wandbruntime‚5703  _runtime‚5703^V@½'©úM RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10879792398336 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196844424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚841514518218752 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111175806 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10556252970496 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 _step‚891 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196834265 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚679319725505536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196834265 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚852070788707840 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10556252970496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 % _timestamp‚1.7844922270798602e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10555912177152 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87395940 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58321391 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10555912177152 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87396210 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85668616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196844424 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29074817 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚679319880995840 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚852070430395904 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111165649 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29074549 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚841514535737344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚668439933107200 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚832700138552320 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111165649 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚10999 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚852070788707840 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85668616 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85668618 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10536925204102 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58321393 _runtime‚5703.654978145 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85668618 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚668439957531648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 144074342409R'RªÿM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10555912177152 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87396210 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85668616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196844424 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29074817 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚679319880995840 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚852070430395904 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111165649 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29074549 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚841514535737344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.<€µÙ³current‚17518592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚668439933107200 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚832700138552320 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111165649 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚10999 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚852070788707840 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85668616 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85668618 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10536925204102 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58321393 _runtime‚5703.654978145 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85668618 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚668439957531648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10879792398336 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196844424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚841514518218752 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111175806 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10556252970496 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 _step‚891 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196834265 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚679319725505536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196834265 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚852070788707840 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10556252970496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 % _timestamp‚1.7844922270798602e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10555912177152 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87395940 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58321391 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720ûroAR2«- _wandbruntime‚5703  _runtime‚5703®«5Ë ¬ƒ 0loss/FRt2/assignment_loss‚1.7422352935798309 3loss/FBt1/assignment_loss‚0.0008764319010691983 1loss/SRqqt1/detection_loss‚0.5886237885553522 % _timestamp‚1.7844922270807927e+09 1loss/SRqqt2/detection_loss‚0.8677867823705053 2loss/SRqqt1/assignment_loss‚0.5216658481782568 2loss/SRqqt2/assignment_loss‚0.5873222092339335 /loss/FRt1/detection_loss‚1.4584491012195027 #loss/total_loss‚8.0740822117297 _runtime‚5703.655482226 .loss/FRt1/assignment_loss‚0.75943783890222 3loss/FBt2/assignment_loss‚7.812339715557597e-05 /loss/FRt2/detection_loss‚0.5028522816235907 /loss/FBt1/detection_loss‚0.4290682136322305 /loss/FBt2/detection_loss‚0.6156862991360518 epoch‚13 trainer/global_step‚10999 _step‚892怭ˆ 1loss/SRqqt2/detection_loss‚0.8677867823705053 2loss/SRqqt1/assignment_loss‚0.5216658481782568 2loss/SRqqt2/assignment_loss‚0.5873222092339335 /loss/FRt1/detection_loss‚1.4584491012195027 #loss/total_loss‚8.0740822117297 _runtime‚5703.655482226 .loss/FRt1/assignment_loss‚0.75943783890222 3loss/FBt2/assignment_loss‚7.812339715557597e-05 /loss/FRt2/detection_loss‚0.5028522816235907 /loss/FBt1/detection_loss‚0.4290682136322305 /loss/FBt2/detection_loss‚0.6156862991360518 epoch‚13 trainer/global_step‚10999 _step‚892 0loss/FRt2/assignment_loss‚1.7422352935798309 3loss/FBt1/assignment_loss‚0.0008764319010691983 1loss/SRqqt1/detection_loss‚0.5886237885553522 % _timestamp‚1.7844922270807927e+09üš 5Êž®‚(:“ ÎáôÒ‡åÃÌ gpu.0.gpu‚79.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚261.393' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.13099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†f%Íg¯‚(:Ü ÎáôÒÝÒ‹Í network.recv‚ 1.534461e+06 disk.nvme1c1n1.in‚0 disk.rbd0.out‚ 0.4697265625$ memory_percent‚96.99337482452393* proc.memory.percent‚17.352581024169922 proc.cpu.threads‚17 network.sent‚ 1.508869e+07 disk.nvme0n1.in‚ 4.3359375% disk.nvme0n1.out‚50832.5966796875 disk.rbd0.in‚0( proc.memory.availableMB‚ 492.60546875 cpu‚26.1278757267905 disk.nvme1n1.in‚0% disk.nvme1n1.out‚19831.0888671875 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚61731.21240234375 disk.dm-1.in‚0! proc.memory.rssMB‚ 2843.046875) disk./.usagePercent‚76.54650125533047$ disk./.usageGB‚1175.180160522461' disk.nvme1c1n1.out‚19831.0888671875 disk.md0.in‚0 disk.md0.out‚ 148.546875" disk.dm-1.out‚8612.99365234375 disk.nvme0c0n1.in‚ 4.3359375' disk.nvme0c0n1.out‚50832.5966796875Úåò¾2°- _wandbruntime‚5727  _runtime‚5727×íԪȱ _runtime‚5727.240869475 _step‚893 & lr-AdamW/pg1‚0.0001790082326602035 & lr-AdamW/pg2‚0.0001790082326602035 trainer/global_step‚11049 % _timestamp‚1.7844922506658983e+09ó2³ÓÍ²Ç _runtime‚5727.240869475 _step‚893 & lr-AdamW/pg1‚0.0001790082326602035 & lr-AdamW/pg2‚0.0001790082326602035 trainer/global_step‚11049 % _timestamp‚1.7844922506658983e+09ý®’kô2³- _wandbruntime‚5727  _runtime‚5727McT'ô'´îO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚855751783296512 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚855751424991744 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚682251631709696 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 197687590 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111651376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚671324897826304 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚11049 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10601510087680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87768095 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10926758307840 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 % _timestamp‚1.7844922506667464e+09 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚845149932422656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10582779969196 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10582440045874 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚855751424991744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58570362 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚671324873401856 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚836298016593840 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58570360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10601510087680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚836297999554480 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87767813 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚845149932422656 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 197697735 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚682251789304320 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚845149914904064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚855751783296512 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚845149914904064 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86036214 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86036214 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29197453 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚855751424991744 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚682251631709696 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 197687590 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111651376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚671324897826304 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚11049 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10601510087680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87768095 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10926758307840 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 % _timestamp‚1.7844922506667464e+09 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚845149932422656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10582779969196 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10582440045874 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚855751424991744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58570362 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚671324873401856 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚836298016593840 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58570360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10601510087680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚836297999554480 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87767813 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚845149932422656 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 197697735 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚682251789304320 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚845149914904064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935þ2 À+2¶- _wandbruntime‚5727  _runtime‚5727–r6…·úM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.onÑ£{¦{$_train_batch_end/active_bytes.all.current‚ 358311936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10927850659328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 _step‚895 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111671408 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 197704967 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87775650 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29200207 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚846954793906070 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 % _timestamp‚1.7844922510209448e+09 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10602439998976 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚855826546672128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚682311473522176 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚846955150868864 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 197704967 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 197715126 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58575443 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10583708192544 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚855826904984064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 197715126 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87775381 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚11049 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86043718 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚855826904984064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111661251 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚845224106673152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10583368269110 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111671408 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚836371442676320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚5727.596104109 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111661251 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10602780792320 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86043718 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29199940 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58575441 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86043716 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚845224124191744 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚671383774158848 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚845224106673152 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10602439998976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚836371425636960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10602780792320W2J'¸ÿM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10583708192544 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚855826904984064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 197715126 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87775381 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚11049 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86043718 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚855826904984064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111661251 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚845224106673152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10583368269110 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111671408 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚836371442676320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚5727.596104109 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111661251 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10602780792320 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86043718 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29199940 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58575441 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86043716 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚845224124191744 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚671383774158848 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚845224106673152 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10602439998976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚836371425636960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10602780792320 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10927850659328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 _step‚895 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111671408 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 197704967 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87775650 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29200207 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚846954793906070 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 % _timestamp‚1.7844922510209448e+09 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10602439998976 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚855826546672128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚682311473522176 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚846955150868864 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 197704967 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 197715126 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58575443ÿקM2¹- _wandbruntime‚5727  _runtime‚5727k½Äºÿ 0loss/FRt1/assignment_loss‚0.5728373376376321 1loss/SRqqt2/assignment_loss‚0.655662568181829 2loss/FBt1/assignment_loss‚0.031217184248345853 %loss/total_loss‚8.270633953642339 _step‚896 epoch‚13 /loss/FRt2/assignment_loss‚1.742428494587777 1loss/FBt2/assignment_loss‚0.10307683577928042 /loss/FRt1/detection_loss‚1.4857423311928242 /loss/FRt2/detection_loss‚0.5333542945928846 1loss/SRqqt2/detection_loss‚0.8449288953074016 /loss/FBt2/detection_loss‚0.6337514068665249 trainer/global_step‚11049 1loss/SRqqt1/detection_loss‚0.5537646750566727 % _timestamp‚1.7844922510218508e+09 _runtime‚ 5727.59661058 0loss/SRqqt1/assignment_loss‚0.51428115670863 /loss/FBt1/detection_loss‚0.5995887734825374g[» »„ 0loss/SRqqt1/assignment_loss‚0.51428115670863 /loss/FBt1/detection_loss‚0.5995887734825374 0loss/FRt1/assignment_loss‚0.5728373376376321 1loss/SRqqt2/assignment_loss‚0.655662568181829 2loss/FBt1/assignment_loss‚0.031217184248345853 %loss/total_loss‚8.270633953642339 _step‚896 epoch‚13 /loss/FRt2/assignment_loss‚1.742428494587777 1loss/FBt2/assignment_loss‚0.10307683577928042 /loss/FRt1/detection_loss‚1.4857423311928242 /loss/FRt2/detection_loss‚0.5333542945928846 1loss/SRqqt2/detection_loss‚0.8449288953074016 /loss/FBt2/detection_loss‚0.6337514068665249 trainer/global_step‚11049 1loss/SRqqt1/detection_loss‚0.5537646750566727 % _timestamp‚1.7844922510218508e+09 _runtime‚ 5727.59661058€åDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86411316 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88148221 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚674268714453504 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112157121 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10974849451520 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10974728864256 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10648378695680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58824410 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚859507541267968 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚839969320717840 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86411316 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚674268690029056 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88147949¹ï¬û'ÅõO [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10629223034204 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112146978 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚848859520877056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10974849451520 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10974728864256 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10648378695680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58824410 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚859507541267968 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚839969320717840 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86411316 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚674268690029056 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88147949 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚848859503358464 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198568437 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86411314 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 % _timestamp‚1.7844922754581559e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112157121 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10648037909504 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚685243563905024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 145011712 trainer/global_step‚11099 _runtime‚5752.033211025 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58824412 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86411316 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88148221 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚674268714453504 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112157121 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198585828 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29326288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112167010 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112156853 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚848933712646144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 trainer/global_step‚11099 _step‚899 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88155508 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚674327566361600 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86418818 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10975687475200 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86418816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198585828 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 _runtime‚5752.389907305 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10649308614144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88155781 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 % _timestamp‚1.7844922758148339e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚848933712646144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112156853 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚840042746800320 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚674327590786048 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10630151257552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚859583021260288 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚685303403035648 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10975812249600 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚850672541095078 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112167010 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10629811334118 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚840042729760960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29326017 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198575669 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10648967820800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10649308614144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚859583021260288 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10648967820800 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198575669 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58829493 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 _runtime‚5752.389907305 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10649308614144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88155781 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149198848 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 % _timestamp‚1.7844922758148339e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚848933712646144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112156853 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚840042746800320 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚674327590786048 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10630151257552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚859583021260288 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚685303403035648 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10975812249600 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚850672541095078 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112167010 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10629811334118 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚840042729760960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29326017 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198575669 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10648967820800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10649308614144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚859583021260288 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10648967820800 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198575669 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58829493 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198585828 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29326288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112167010 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112156853 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚848933712646144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 trainer/global_step‚11099 _step‚899 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88155508 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚674327566361600 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86418818 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10975687475200 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935ƒ)ŠÖ2É- _wandbruntime‚5752  _runtime‚5752Ú(Ö Ê… .loss/FRt1/detection_loss‚1.323974437813234 1loss/SRqqt2/detection_loss‚0.8237949269639916 trainer/global_step‚11099 % _timestamp‚1.7844922758157706e+09 /loss/FRt2/assignment_loss‚1.815603282742613 %loss/total_loss‚7.946459591730205 1loss/SRqqt1/assignment_loss‚0.488529866890967 4loss/FBt2/assignment_loss‚0.00011599674626220728 /loss/FRt2/detection_loss‚0.5131639678050227 1loss/SRqqt1/detection_loss‚0.5915881335449541 /loss/FBt2/detection_loss‚0.6848833974864748 0loss/FRt1/assignment_loss‚0.7055608422859855 3loss/FBt1/assignment_loss‚0.0022643475838753702 /loss/FBt1/detection_loss‚0.3653911339759361 epoch‚13 _runtime‚5752.390399686 _step‚900 2loss/SRqqt2/assignment_loss‚0.6315892578908892 Ì•.MËŠ 0loss/FRt1/assignment_loss‚0.7055608422859855 3loss/FBt1/assignmuiŽÃent_loss‚0.0022643475838753702 /loss/FBt1/detection_loss‚0.3653911339759361 epoch‚13 _runtime‚5752.390399686 _step‚900 2loss/SRqqt2/assignment_loss‚0.6315892578908892 .loss/FRt1/detection_loss‚1.323974437813234 1loss/SRqqt2/detection_loss‚0.8237949269639916 trainer/global_step‚11099 % _timestamp‚1.7844922758157706e+09 /loss/FRt2/assignment_loss‚1.815603282742613 %loss/total_loss‚7.946459591730205 1loss/SRqqt1/assignment_loss‚0.488529866890967 4loss/FBt2/assignment_loss‚0.00011599674626220728 /loss/FRt2/detection_loss‚0.5131639678050227 1loss/SRqqt1/detection_loss‚0.5915881335449541 /loss/FBt2/detection_loss‚0.6848833974864748„¹š*gÌ‚(:Ü ûáôÒ÷î÷Ë disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚19867.7490234375 disk.nvme1n1.in‚0! disk.dm-0.out‚61995.505859375& proc.memory.availableMB‚ 492.453125! proc.memory.rssMB‚ 2843.046875 network.sent‚ 1.5219488e+07) disk./.usagePercent‚76.54656213146335 disk.nvme0n1.in‚ 4.3359375% disk.nvme1n1.out‚19867.7490234375 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375$ memory_percent‚96.99430465698242* proc.memory.percent‚17.352581024169922 network.recv‚ 1.546173e+06$ disk./.usageGB‚1175.181095123291" disk.dm-1.out‚8613.65771484375 disk.rbd0.out‚ 0.4697265625 disk.dm-1.in‚0( disk.nvme0c0n1.out‚51063.47998046875& disk.nvme0n1.out‚51063.47998046875 disk.md0.out‚ 149.765625 cpu‚26.12831882107329 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375h¾¼‘Í‚(:† ûáôÒªî«Ì gpu.0.gpu‚63.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚252.39' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚84.13 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨5X΂(:Í ŠâôÒïí×Ë! disk.nvme0n1.out‚ 51068.640625 disk.md0.in‚0 disk.dm-1.in‚0" disk.dm-1.out‚8613.72802734375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚62000.51708984375 disk.rbd0.out‚ 0.4697265625$ memory_percent‚96.99430465698242& proc.memory.availableMB‚ 492.453125 network.recv‚ 1.550694e+06! disk.nvme1n1.out‚ 19868.140625 disk.md0.out‚ 149.9765625! proc.memory.rssMB‚ 2843.046875* proc.memory.percent‚17.352581024169922 cpu‚26.12902516474018 network.sent‚ 1.5268015e+07# disk.nvme1c1n1.out‚ 19868.140625 disk.nvme1n1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚76.54656213146335$ disk./.usageGB‚1175.181095123291# disk.nvme0c0n1.out‚ 51068.640625wüíŸÏ‚(:” ŠâôÒÏÈÌ gpu.0.gpu‚39.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚118.817' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.605666666666664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æX¤¹2Ð- _wandbruntime‚5776  _runtime‚5776i’wÜÊÑÄ ' lr-AdamW/pg2‚0.00017838292637551258 trainer/global_step‚11149 % _timestamp‚1.7844922996749842e+09 _runtime‚5776.249896591 _step‚901 ' lr-AdamW/pg1‚0.00017838292637551258Ñ·ò/ÏÒÉ _runtime‚5776.249896591 _step‚901 ' lr-AdamW/pg1‚0.00017838292637551258 ' lr-AdamW/pg2‚0.00017838292637551258 trainer/global_step‚11149 % _timestamp‚1.7844922996749842e+09…’/=ó2Ó- _wandbruntime‚5776  _runtime‚5776ã¿'òö'ÔðO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199439139 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88528425 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚863263657544192 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112652723 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚688235346885120 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11022815804416 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚852569091812864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 % _timestamp‚1.7844922996762562e+09 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚843640624841840 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚863263657544192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚677212531080704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚854316290941052 _step‚902 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10694565731328 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚677212506656256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59078462 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚854315933978370 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚852569091812864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚852569109331456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10694565731328 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚863264015848960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚852569109331456 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86786414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112652723 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 199428994 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88528699 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10675326175890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 _runtime‚5776.251079433 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86786414 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935§ ´Õû'ÕõO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59078462 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚854315933978370 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚852569091812864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚852569109331456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10694565731328 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚863264015848960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚852569109331456 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86786414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112652723 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 199428994 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88528699 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10675326175890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 _runtime‚5776.251079433 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86786414 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199439139 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88528425 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚863263657544192 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112652723 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚688235346885120 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11022815804416 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚852569091812864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 % _timestamp‚1.7844922996762562e+09 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚843640624841840 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚863263657544192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚677212531080704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚854316290941052 _step‚902 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10694565731328 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚677212506656256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0†¶–,2Ö- _wandbruntime‚5776  _runtime‚5776|Vç1Ó ×úM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88536291 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚854390645246880 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚677271407413248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88536021 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚677271382988800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86793918 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10695495642624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚863339137536512 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10695495642624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29452748 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11023646236160 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 _step‚903 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 199456530 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11023773107712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112652455 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚852643283581952 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10695836435968 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚688295029224960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29452480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199456530 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚863339137536512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10676594322560 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚688295180520960 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10676254399126 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_endb¤-/active.large_pool.freed‚86793916 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59083543 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7844923000335271e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675{6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10676594322560 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚688295180520960 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10676254399126 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86793916 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59083543 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7844923000335271e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88536291 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚854390645246880 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚677271407413248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88536021 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151296000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚677271382988800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86793918 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10695495642624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚863339137536512 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10695495642624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29452748 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11023646236160 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 _step‚903 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 199456530 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11023773107712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112652455 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚852643283581952 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10695836435968 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚688295029224960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29452480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199456530 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240‡ë—2Ù- _wandbruntime‚5776  _runtime‚5776«þEm Ú… %loss/total_loss‚8.680680245982266 % _timestamp‚1.7844923000344844e+09 1loss/SRqqt1/detection_loss‚0.6319341337470377 /loss/FBt1/detection_loss‚0.5293771657420115 /loss/FBt2/detection_loss‚1.0058353916845388 epoch‚13 0loss/FRt2/assignment_loss‚1.6625572488516625 2loss/SRqqt1/assignment_loss‚0.5467013856928361 _step‚904 2loss/SRqqt2/assignment_loss‚0.5955778518069077 trainer/global_step‚11149 _runtime‚5776.609108808 0loss/FRt1/assignment_loss‚0.7770638251203602 2loss/FBt1/assignment_loss‚0.003524277441536149 3loss/FBt2/assignment_loss‚0.0034250445476860453 /loss/FRt1/detection_loss‚1.4197001092341113 .loss/FRt2/detection_loss‚0.562994818960718 1loss/SRqqt2/detection_loss‚0.9419889931528608>{©ÄÛŠ 2loss/SRqqt2/assignment_loss‚0.5955778518069077 trainer/global_step‚11149 _runtime‚5776.609108808 0loss/FRt1/assignment_loss‚0.7770638251203602 2loss/FBt1/assignment_loss‚0.003524277441536149 3loss/FBt2/assignment_loss‚0.0034250445476860453 /loss/FRt1/detection_loss‚1.4197001092341113 .loss/FRt2/detection_loss‚0.562994818960718 1loss/SRqqt2/detection_loss‚0.9419889931528608 %loss/total_loss‚8.680680245982266 % _timestamp‚1.7844923000344844e+09 1loss/SRqqt1/detection_loss‚0.6319341337470377 /loss/FBt1/detection_loss‚0.5293771657420115 /loss/FBt2/detection_loss‚1.0058353916845388 epoch‚13 0loss/FRt2/assignment_loss‚1.6625572488516625 2loss/SRqqt1/assignment_loss‚0.5467013856928361 _step‚904ˆvÇDXkÜ‚(:à ™âôÒ´©ÄË proc.cpu.threads‚17% disk.nvme0n1.out‚51074.5126953125 disk.md0.out‚ 150.2109375# disk.dm-0.out‚62061.21240234375" disk.dm-1.out‚8613.91162109375 disk.rbd0.out‚ 0.4697265625( proc.memory.availableMB‚ 491.87890625 network.sent‚ 1.5273892e+07% disk./.usageGB‚1175.2450942993164 disk.nvme0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0$ memory_percent‚96.99780941009521 cpu‚26.12951155098132' disk.nvme1c1n1.out‚19923.6728515625 disk.md0.in‚0' disk.nvme0c0n1.out‚51074.5126953125* proc.memory.percent‚17.352581024169922 disk.dm-0.in‚ 4.3359375 network.recv‚ 1.554227e+06) disk./.usagePercent‚76.55073077995873 disk.nvme1n1.in‚0% disk.nvme1n1.out‚19923.6728515625 disk.nvme0c0n1.in‚ 4.3359375! proc.memory.rssMB‚ 2843.046875ò2X…“Ý‚(:ˆ ™âôÒ×™¢Ì gpu.0.gpu‚94.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚251.997' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚83.999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§12Þ- _wandbruntime‚5801  _runtime‚5801 uá¥ÊßÄ ' lr-AdamW/pg1‚0.00017806854188737048 ' lr-AdamW/pg2‚0.00017806854188737048 trainer/global_step‚11199 % _timestamp‚1.7844923247169838e+09 _runtime‚5801.292718067 _step‚905qÛÄÏÏàÉ ' lr-AdamW/pg1‚0.00017806854188737048 ' lr-AdamW/pg2‚0.00017806854188737048 trainer/global_step‚11199 % _timestamp‚1.7844923247169838e+09 _runtime‚5801.292718067 _step‚905‰ái Ê2á- _wandbruntime‚5801  _runtime‚58014Ë|3ö'âðO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚867020132125184 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87161516 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29577167 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 _step‚906 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 200299696 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚867020132125184 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88909679 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚847311928965840 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚847311911926480 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200309841 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚680156347707904 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚856278697785856 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29576884 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113138182 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 145011712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚680156323283456 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 trainer/global_step‚11199 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 _runtime‚5801.292778337 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚856278680267264 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚867019773820416 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113138182 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚691227021825536 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10741093553152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚867019773820416 % _timestamp‚1.7844923247177591e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10741434339328 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88909394 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935·-þ5ãõO QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113138182 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚691227021825536 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10741093553152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚867019773820416 % _timestamp‚1.7844923247177591e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10741434339328 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88909394 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚867020132125184 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87161516 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29577167 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 _step‚906 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 200299696 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚867020132125184 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88909679 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚847311928965840 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚847311911926480 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200309841 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚680156347707904 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚856278697785856 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29576884 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113138182 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 145011712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚680156323283456 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 trainer/global_step‚11199 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 _runtime‚5801.292778337 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 LBDeviceStatsMonitorú6LÆ .on_train_batch_start/allocation.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚856278680267264 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚867019773820416 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0Š}ð¦2ä- _wandbruntime‚5801  _runtime‚5801>‰ƒ?'åúM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59337591 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚680215199616000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29579705 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10742023464448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200327232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 trainer/global_step‚11199 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚867095253812736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚680215224040448 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10742364257792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200317073 _step‚907 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113148057 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚847385338008960 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200327232 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87169018 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113158214 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 _runtime‚5801.665472423 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚691286855511552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88917298 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚856352889554944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29579418 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚856352889554944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10742364257792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚867095253812736 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87169018 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88917009 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935C¿['æÿM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87169018 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88917009 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59337591 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚680215199616000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29579705 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10742023464448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200327232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 trainer/global_step‚11199 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚867095253812736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚680215224040448 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10742364257792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200317073 _step‚907 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113148057 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚847385338008960 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200327232 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87169018 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113158214 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 _runtime‚5801.665472423 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚691286855511552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126871552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88917298 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚856352889554944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29579418 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚856352889554944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10742364257792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚867095253812736 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260‹ ‡?_2ç- _wandbruntime‚5801  _runtime‚5801Üh‘¼ è… 2loss/SRqqt2/assignment_loss‚0.6229239702153684 /loss/FRt2/detection_loss‚0.5648327907786028 %loss/total_loss‚7.808367021377961 0loss/FRt1/assignment_loss‚0.7679792363469193 0loss/FRt2/assignment_loss‚1.6796109035159603 2loss/FBt1/assignment_loss‚0.009948556200600223 /loss/FBt1/detection_loss‚0.2844147431605961 0loss/FBt2/detection_loss‚0.39662200631573796 % _timestamp‚1.7844923250914092e+09 _runtime‚5801.666055924 1loss/FBt2/assignment_loss‚0.00934528385589205 .loss/FRt1/detection_loss‚1.507618834280477 1loss/SRqqt1/detection_loss‚0.6608437846791644 trainer/global_step‚11199 _step‚908 3loss/SRqqt1/assignment_loss‚0.38997752693924875 1loss/SRqqt2/detection_loss‚0.9142493850893944 epoch‚13ÊJ|éŠ /loss/FBt1/detection_loss‚0.2844147431605961 0loss/FBt2/detection_loss‚0.39662200631573796 % _timestamp‚1.7844923250914092e+09 _runtime‚5801.666055924 1loss/FBt2/assignment_loss‚0.00934528385589205 .loss/FRt1/detection_loss‚1.507618834280477 1loss/SRqqt1/detection_loss‚0.6608437846791644 trainer/global_step‚11199 _step‚908 3loss/SRqqt1/assignment_loss‚0.38997752693924875 1loss/SRqqt2/detection_loss‚0.9142493850893944 epoch‚13 2loss/SRqqt2/assignment_loss‚0.6229239702153684 /loss/FRt2/detection_loss‚0.5648327907786028 %loss/total_loss‚7.808367021377961 0loss/FRt1/assignment_loss‚0.7679792363469193 0loss/FRt2/assignment_loss‚1.6796109035159603 2loss/FBt1/assignment_loss‚0.009948556200600223ŒäÎÞï’ê‚(:‡ ¨âôÒ‘å’Ì gpu.0.gpu‚34.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚257.382' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚85.794 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü|‚ögë‚(:Ü ¨âôÒÆšƒÎ disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚19925.20556640625 disk.rbd0.out‚ 0.4697265625$ memory_percent‚96.99804782867432! proc.memory.rssMB‚ 2843.046875* proc.memory.percent‚17.352581024169922 network.sent‚ 1.5336401e+07 disk.md0.in‚0 disk.md0.out‚150.875 disk.dm-0.in‚ 4.3359375 disk.rbd0.in‚0 proc.cpu.threads‚17' disk.nvme0c0n1.out‚51088.5771484375( disk.nvme1c1n1.out‚19925.20556640625 disk.nvme1n1.in‚0! disk.dm-0.out‚62075.294921875( proc.memory.availableMB‚ 491.83984375 network.recv‚ 1.558928e+06) disk./.usagePercent‚76.55072978606269% disk./.usageGB‚1175.2450790405273% disk.nvme0n1.out‚51088.5771484375 disk.dm-1.in‚0" disk.dm-1.out‚8614.01708984375 cpu‚26.13031423872767 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375…§]"fì‚(:Û ·âôÒæÉêË network.sent‚ 1.5399479e+07) disk./.usagePercent‚76.56615654355714" disk.dm-1.out‚8614.20654296875( disk.nvme0c0n1.out‚51246.73681640625 disk.nvme1n1.in‚0 disk.dm-0.in‚ 4.3359375* proc.memory.percent‚17.355680465698242 cpu‚26.13814183361988 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375 proc.cpu.threads‚17 network.recv‚ 1.562711e+06 disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625$ disk.nvme1c1n1.out‚ 20011.3359375 disk.md0.in‚0 disk.md0.out‚ 151.13671875" proc.memory.rssMB‚ 2843.5546875$ disk./.usageGB‚1175.481918334961 disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚51246.73681640625 disk.nvme1c1n1.in‚0" disk.nvme1n1.out‚ 20011.3359375! disk.dm-0.out‚62318.833984375$ memory_percent‚97.00119495391846( proc.memory.availableMB‚ 491.32421875'ˆží‚(:“ ·âôÒ‡ä¡Ì gpu.0.gpu‚55.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚267.967' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.32233333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«ó&2î- _wandbruntime‚5828  _runtime‚5828 „@",ïÞ{ particle/accuracy_1_of_1‚1 jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6530852913856506 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9945598840713501 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9922288060188293 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,"Purity/1FRt0SRqqt2FBt/event_purity‚0.875 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6701388955116272 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3052694797515869 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7064575552940369 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9975169897079468 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7160150408744812 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39443978667259216 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9949747920036316 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9954544901847839 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.253315806388855 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_3‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9976615905761719 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9067913889884949 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4135802090167999 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24232244491577148 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4545454680919647 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14151771366596222 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6638392210006714 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7320242524147034 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9952380657196045 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7861611247062683 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.022274186834692955 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1428571492433548 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9982993602752686 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6620538234710693 +jet/accuracy_3_of_3‚0.07754868268966675 particle/accuracy_1_of_2‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1428571492433548 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.25001922249794006 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9956774115562439 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12690111994743347 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22315232455730438 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.17031250894069672 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 6"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8599893450737 'particle/f_score‚0.6515418291091919 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.758155345916748 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9983088374137878 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6928333044052124 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6638392210006714 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 *jet/accuracy_1_of_4‚0.9685280323028564 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9561665654182434 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6583904027938843 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14196594059467316 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23950910568237305 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7515712380409241 (particle/accuracy‚0.8530389666557312 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.26175227761268616 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925780296325684 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9975169897079468 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.058057188987731934 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3613627552986145 jet/accuracy_3_of_5‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8914393186569214 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_5_of_5‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.022274186834692955 +jet/accuracy_3_of_4‚0.09238354861736298 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.903508722782135 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.31791552901268005 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.692686915397644 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.055152904242277145 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.31029507517814636 ,jet/accuracy_4_of_4‚0.014682539738714695 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 )loss/val_total_loss‚8.17468434093315‰¯U¸'1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7418621778488159 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 *jet/accuracy_1_of_2‚0.7935737371444702 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.949999988079071 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9955229759216309 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24080072343349457 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2787255644798279 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 _step‚909 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6587207913398743 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9932540059089661 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10000000149011612 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9844576716423035 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5119265913963318 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.029104042798280716 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6751241087913513 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9912595152854919 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.043772511184215546 *jet/accuracy_1_of_3‚0.9315702319145203 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23642314970493317 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10980617254972458 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9888187646865845 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9958263635635376 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5739222168922424 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 % _timestamp‚1.7844923523586824e+09 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39725011587142944 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46246829628944397 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_4_of_5‚1 jet/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6485355496406555 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.47142696380615234 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970107078552246 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03112584352493286 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9253242015838623 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3819444477558136 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7234504222869873 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.298658549785614 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9085213541984558 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7096878290176392 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.29818037152290344 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6533236503601074 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8038380146026611 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7545191049575806 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4650000035762787 particle/accuracy_2_of_2‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 particle/accuracy_3_of_5‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5242145657539368 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 6"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8599893450737 _runtime‚5828.934106427 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.36845725774765015 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6250060796737671 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23229992389678955 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.9081196784973145 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7120060920715332 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.9012515544891357 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.16890953481197357 particle/accuracy_2_of_3‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5389385223388672 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.31822842359542847 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.33458012342453003 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04956737160682678 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5991382002830505 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7442654371261597 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 /particle/accuracy_3_of_4‚0.9952380657196045 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.027514919638633728 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6716761589050293 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7290472388267517 *jet/accuracy_2_of_4‚0.7994190454483032 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8025640249252319 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7685839533805847 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.30726951360702515 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5711964964866638 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6508517265319824 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7299691438674927 +particle/sensitivity‚0.6398642063140869 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10000000149011612 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_5‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.24206404387950897 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9967128038406372 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6771224141120911 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5897027254104614 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4422610104084015 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5572916865348816 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.028676077723503113 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8016480207443237 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9916865229606628 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8704606890678406 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6616107225418091 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.30726951360702515 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7699999809265137 particle/accuracy_1_of_3‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3610580265522003 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966694712638855 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 /particle/accuracy_4_of_4‚0.9344276189804077 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 jet/accuracy_4_of_5‚0 epoch‚13 trainer/global_step‚11213 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7621443867683411 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 +particle/specificity‚0.9113360047340393 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9285714030265808 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9885275959968567 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3169211149215698 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7925829291343689 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6910170316696167 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15677106380462646 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 6validation_average_jet_accuracy‚0.3980817198753357 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0944729894399643 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22750325500965118 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26451465487480164 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7699999809265137‘çÔáé=ðã{ jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6530852913856506 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9945598840713501 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9922288060188293 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,"Purity/1FRt0SRqqt2FBt/event_purity‚0.875 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6701388955116272 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3052694797515869 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7064575552940369 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9975169897079468 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7160150408744812 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39443978667259216 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9949747920036316 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9954544901847839 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.253315806388855 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_3‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9976615905761719 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9067913889884949 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4135802090167999 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24232244491577148 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4545454680919647 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14151771366596222 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6638392210006714 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7320242524147034 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9952380657196045 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7861611247062683 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.022274186834692955 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1428571492433548 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9982993602752686 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6620538234710693 +jet/accuracy_3_of_3‚0.07754868268966675 particle/accuracy_1_of_2‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1428571492433548 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.25001922249794006 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9956774115562439 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12690111994743347 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22315232455730438 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.17031250894069672 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 6"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8599893450737 'particle/f_score‚0.6515418291091919 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.758155345916748 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9983088374137878 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6928333044052124 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6638392210006714 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 *jet/accuracy_1_of_4‚0.9685280323028564 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9561665654182434 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6583904027938843 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14196594059467316 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23950910568237305 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7515712380409241 (particle/accuracy‚0.8530389666557312 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.26175227761268616 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925780296325684 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9975169897079468 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.058057188987731934 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3613627552986145 jet/accuracy_3_of_5‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8914393186569214 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_5_of_5‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.022274186834692955 +jet/accuracy_3_of_4‚0.09238354861736298 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.903508722782135 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.31791552901268005 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.692686915397644 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.055152904242277145 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.31029507517814636 ,jet/accuracy_4_of_4‚0.014682539738714695 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 )loss/val_total_loss‚8.174684340933151 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7418621778488159 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 *jet/accuracy_1_of_2‚0.7935737371444702 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.949999988079071 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9955229759216309 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24080072343349457 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2787255644798279 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 _step‚909 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6587207913398743 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9932540059089661 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10000000149011612 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9844576716423035 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5119265913963318 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.029104042798280716 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6751241087913513 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9912595152854919 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.043772511184215546 *jet/accuracy_1_of_3‚0.9315702319145203 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.23642314970493317 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10980617254972458 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9888187646865845 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9958263635635376 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5739222168922424 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 % _timestamp‚1.7844923523586824e+09 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39725011587142944 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46246829628944397 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_4_of_5‚1 jet/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6485355496406555 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.47142696380615234 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970107078552246 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03112584352493286 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9253242015838623 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3819444477558136 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7234504222869873 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.298658549785614 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9085213541984558 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7096878290176392 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.29818037152290344 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6533236503601074 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8038380146026611 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7545191049575806 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4650000035762787 particle/accuracy_2_of_2‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 particle/accuracy_3_of_5‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5242145657539368 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 6"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8599893450737 _runtime‚5828.934106427 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.36845725774765015 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6250060796737671 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23229992389678955 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.9081196784973145 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7120060920715332 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.9012515544891357 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.16890953481197357 particle/accuracy_2_of_3‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5389385223388672 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.31822842359542847 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.33458012342453003 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04956737160682678 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5991382002830505 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7442654371261597 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 /particle/accuracy_3_of_4‚0.9952380657196045 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.027514919638633728 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6716761589050293 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7290472388267517 *jet/accuracy_2_of_4‚0.7994190454483032 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8025640249252319 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7685839533805847 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.30726951360702515 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5711964964866638 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6508517265319824 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7299691438674927 +particle/sensitivity‚0.6398642063140869 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10000000149011612 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_5‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.24206404387950897 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9967128038406372 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6771224141120911 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5897027254104614 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4422610104084015 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5572916865348816 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.028676077723503113 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8016480207443237 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9916865229606628 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8704606890678406 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6616107225418091 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.30726951360702515 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7699999809265137 particle/accuracy_1_of_3‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3610580265522003 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966694712638855 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 /particle/accuracy_4_of_4‚0.9344276189804077 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 jet/accuracy_4_of_5‚0 epoch‚13 trainer/global_step‚11213 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7621443867683411 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 +particle/specificity‚0.9113360047340393 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9285714030265808 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9885275959968567 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3169211149215698 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7925829291343689 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6910170316696167 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15677106380462646 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 6validation_average_jet_accuracy‚0.3980817198753357 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0944729894399643 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22750325500965118 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26451465487480164 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7699999809265137 particle/accuracy_1_of_1‚1á™—’bñ‚(:× ÆâôÒÊÜÿË network.sent‚ 1.5415387e+07 network.recv‚ 1.566059e+06% disk./.usageGB‚1175.4828567504883" disk.dm-0.out‚62324.1220703125 disk.dm-1.in‚0" disk.dm-1.out‚8859.67919921875 disk.nvme0n1.in‚ 4.3359375& proc.memory.availableMB‚ 739.140625 disk.nvme1c1n1.in‚0! disk.nvme1n1.out‚ 20257.515625 disk.md0.in‚0$ memory_percent‚95.48864364624023" proc.memory.rssMB‚ 2843.9609375( proc.memory.percent‚17.3581600189209 cpu‚26.144838615625634# disk.nvme1c1n1.out‚ 20257.515625 disk.nvme1n1.in‚0 disk.md0.out‚ 151.5546875 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375 proc.cpu.threads‚17) disk./.usagePercent‚76.56621766816403 disk.dm-0.in‚ 4.3359375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚51252.20849609375& disk.nvme0n1.out‚51252.20849609375”ÝrÂò‚(:’ ÆâôÒæØ°Ì gpu.0.gpu‚78.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚52.0 gpu.0.powerWatts‚155.2' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚51.733333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤¦Œ2ó- _wandbruntime‚5847  _runtime‚58474Ò>Èô % _timestamp‚1.7844923704514835e+09 _runtime‚5847.028403947 _step‚910 & lr-AdamW/pg1‚0.0001777530098765514 & lr-AdamW/pg2‚0.0001777530098765514 trainer/global_step‚11249{$ôÍõÇ _step‚910 & lr-AdamW/pg1‚0.0001777530098765514 & lr-AdamW/pg2‚0.0001777530098765514 trainer/global_step‚11249 % _timestamp‚1.7844923704514835e+09 _runtime‚5847.028403947Ž8 S2ö- _wandbruntime‚5847  _runtime‚5847Tm2×(÷ðO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113807497 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚872747396805120 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚684693026002432 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10812478216192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚872747755109888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚695836606090752 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497472 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚863693346486378 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29800511 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59773042 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚911 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚861935276893696 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113797354 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 201575132 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10792702309376 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87777780 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚872747396805120 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11143555663872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚11249 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10812478216192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batœÒ£Îch_start/segment.large_pool.peak‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 % _timestamp‚1.7844923704520476e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚872747755109888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87777778 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 _runtime‚5847.028457717 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚89573838 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113797354 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚861935259375104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87777780 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113807497 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚684693050426880 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚695836450593280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 201585277 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10812137430016 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚863692989523696 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10793042232698 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚852900287214320 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚861935276893696 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10812137430016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚89573551 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59773040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11143424590848 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚872747396805120 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11143555663872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚11249 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10812478216192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 % _timestamp‚1.7844923704520476e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚872747755109888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87777778 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 _runtime‚5847.028457717 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚89573838 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113797354 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚861935259375104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87777780 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113807497 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚684693050426880 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚695836450593280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 201585277 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10812137430016 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚863692989523696 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10793042232698 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚852900287214320 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚861935276893696 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10812137430016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚89573551 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59773040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11143424590848 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚872747396805120 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚684693026002432 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10812478216192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚872747755109888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚695836606090752 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497472 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚863693346486378 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29800511 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59773042 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚911Ý!€è2ù- _wandbruntime‚5847  _runtime‚5847ЉôÎ'úúM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚862009468662784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚298 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11144382380544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59778121 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚89581486 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113807229 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚872822876797440 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87785280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚852973730336160 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚862009451144192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11144511349248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 201602668 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 201592509 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚695896284715520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113817386 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚684751926759424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚863767700792206 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚863767343829412 trainer/global_step‚11249 % _timestamp‚1.7844923711249876e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 201602668 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚872822518485504 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113807229 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚300 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10813408134656 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10813408134656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29803363 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87785282 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚862009468662784 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10813067341312 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚695896438108672 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚862009451144192 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89581186 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720?FmóâûÿM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚863767700792206 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚863767343829412 trainer/global_step‚11249 % _timestamp‚1.7844923711249876e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 201602668 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚872822518485504 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113807229 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚300 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10813408134656 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10813408134656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29803363 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87785282 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚862009468662784 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10813067341312 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚695896438108672 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚862009451144192 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89581186 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚862009468662784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚298 ^KDeviceStatsMo/ãP¼# nitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11144382380544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59778121 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚89581486 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113807229 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚872822876797440 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87785280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚852973730336160 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚862009451144192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11144511349248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 201602668 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 201592509 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚695896284715520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113817386 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚684751926759424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0A¸+72ü- _wandbruntime‚5847  _runtime‚5847ô"M¤ýþ /loss/FBt1/detection_loss‚0.6362346264418878 .loss/FBt2/detection_loss‚0.989308460216437 %loss/total_loss‚8.563729873801146 _runtime‚ 5847.70088498 epoch‚14 _step‚913 /loss/FRt1/assignment_loss‚0.746748116070574 0loss/FRt2/assignment_loss‚1.5799552106088208 1loss/FBt2/assignment_loss‚0.06369229145950937 /loss/FRt2/detection_loss‚0.5708318867864968 $ _timestamp‚1.784492371126041e+09 2loss/SRqqt1/assignment_loss‚0.5275092135024629 1loss/SRqqt1/detection_loss‚0.6880931774973081 0loss/SRqqt2/detection_loss‚0.823169436087952 trainer/global_step‚11249 2loss/SRqqt2/assignment_loss‚0.5153215451053217 1loss/FBt1/assignment_loss‚0.03273526790017814 /loss/FRt1/detection_loss‚1.3901306421241977i©ï4 þƒ epoch‚14 _step‚913 /loss/FRt1/assignment_loss‚0.746748116070574 0loss/FRt2/assignment_loss‚1.5799552106088208 1loss/FBt2/assignment_loss‚0.06369229145950937 /loss/FRt2/detection_loss‚0.5708318867864968 $ _timestamp‚1.784492371126041e+09 2loss/SRqqt1/assignment_loss‚0.5275092135024629 1loss/SRqqt1/detection_loss‚0.6880931774973081 0loss/SRqqt2/detection_loss‚0.823169436087952 trainer/global_step‚11249 2loss/SRqqt2/assignment_loss‚0.5153215451053217 1loss/FBt1/assignment_loss‚0.03273526790017814 /loss/FRt1/detection_loss‚1.3901306421241977 /loss/FBt1/detection_loss‚0.6362346264418878 .loss/FBt2/detection_loss‚0.989308460216437 %loss/total_loss‚8.563729873801146 _runtime‚ 5847.70088498‘h]åßkÿ‚(:à ÕâôÒÿüåË disk.nvme0c0n1.in‚ 4.3359375( disk.nvme1c1n1.out‚20258.15966796875$ memory_percent‚95.49055099487305 cpu‚26.14519476538255 network.sent‚ 1.5472463e+07 network.recv‚ 1.569918e+06 disk.md0.in‚0 disk.nvme0n1.in‚ 4.3359375( proc.memory.percent‚17.3581600189209 disk.dm-0.in‚ 4.3359375! disk.dm-1.out‚8859.8291015625 disk.rbd0.out‚ 0.4697265625( disk.nvme0c0n1.out‚51256.75341796875& proc.memory.availableMB‚ 738.828125" proc.memory.rssMB‚ 2843.9609375 proc.cpu.threads‚17) disk./.usagePercent‚76.56723169060604% disk./.usageGB‚1175.4984245300293& disk.nvme1n1.out‚20258.15966796875 disk.md0.out‚ 151.92578125" disk.dm-0.out‚62328.3447265625 disk.dm-1.in‚0& disk.nvme0n1.out‚51256.75341796875 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.rbd0.in‚01 € ‚(:’ ÕâôÒ‰ôÌ gpu.0.gpu‚22.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚249.592' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚83.19733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í”Jh ‚(:Ý äâôÒ¶áË% disk./.usageGB‚1175.4983863830566 disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚51268.95166015625 disk.nvme1n1.in‚0# disk.dm-0.out‚62341.17919921875 disk.rbd0.in‚0 proc.cpu.threads‚17 network.sent‚ 1.5524153e+07 disk.md0.out‚ 152.2421875 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 4.3359375( disk.nvme0c0n1.out‚51268.95166015625( proc.memory.availableMB‚ 739.05859375" proc.memory.rssMB‚ 2843.9609375 disk.nvme1c1n1.in‚0! disk.dm-1.out‚9105.3642578125 disk.rbd0.out‚ 0.4697265625$ memory_percent‚95.48914432525635 cpu‚26.144314347971992 network.recv‚ 1.574291e+06% disk.nvme1c1n1.out‚20505.01953125# disk.nvme1n1.out‚20505.01953125 disk.md0.in‚0( proc.memory.percent‚17.3581600189209) disk./.usagePercent‚76.56722920586593ämž‚ ‚(:“ äâôÒóî˜Ì gpu.0.gpu‚70.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚263.203' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.73433333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06J†î2ƒ - _wandbruntime‚5870  _runtime‚5870 ~åxÈ„  % _timestamp‚1.7844923944214005e+09 _runtime‚5870.996365722 _step‚914 & lr-AdamW/pg1‚0.0001774363353969409 & lr-AdamW/pg2‚0.0001774363353969409 trainer/global_step‚11299°¥UFÍ… Ç _step‚914 & lr-AdamW/pg1‚0.0001774363353969409 & lr-AdamW/pg2‚0.0001774363353969409 trainer/global_step‚11299 % _timestamp‚1.7844923944214005e+09 _runtime‚5870.996365722’ªÓ-12† - _wandbruntime‚5870  _runtime‚5870ôÔºiö'‡ ðO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114303099 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚687636867054080 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88152880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚89954243 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88152878 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚865644865348096 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10858665251840 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11191490439680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10859006038016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29927151 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚698828357493760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚876503871386112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚89953949 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚856571591338320 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29926859 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚11299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88152878 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10858665251840 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10839485297706 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10859006038016 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚865644847829504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114292956 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚865644847829504 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147108864 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚867411093675386 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚856571608377680 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 % _timestamp‚1.7844923944222577e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚865644865348096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10839145374384 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 _step‚915 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202455979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114303099 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122684416 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚876503513081344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88152880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114292956 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11191367755264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚876503871386112 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚687636842629632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 _runtime‚5870.997114813 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60027092 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚867410736712704 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚876503513081344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60027090 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0Þ¬>´û'ˆ õO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 % _timestamp‚1.7844923944222577e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚865644865348096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10839145374384 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 _step‚915 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202455979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114303099 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122684416 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚876503513081344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88152880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114292956 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11191367755264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚876503871386112 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚687636842629632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 _runtime‚5870.997114813 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60027092 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚867410736712704 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚876503513081344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60027090 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114303099 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚687636867054080 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88152880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚89954243 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88152878 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚865644865348096 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10858665251840 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11191490439680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10859006038016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29927151 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚698828357493760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚876503871386112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚89953949 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚856571591338320 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29926859 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚11299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88152878 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10858665251840 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10839485297706 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10859006038016 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚865644847829504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114292956 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚865644847829504 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147108864 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚867411093675386 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚856571608377680“ŸtÀ˜2‰ - _wandbruntime‚5871  _runtime‚5871¥¢ £Š úM PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89961526 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202473370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114302831 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10859935956480 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60032171 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚687695718962176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚865719039598592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202473370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10859935956480 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114302831 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88160380 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚856645034460160 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMo›v•ünitor.on_train_batch_end/active.small_pool.allocated‚ 114312988 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚865719057117184 % _timestamp‚1.7844923947798696e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚865719057117184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202463211 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29929355 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚856645017420800 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚698888193646592 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚876578993073664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88160382 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10859595163136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10840073597620 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚916 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10840413521054 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0Êlcò'‹ ÿM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚865719039598592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202473370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10859935956480 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114302831 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88160380 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚856645034460160 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114312988 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚865719057117184 % _timestamp‚1.7844923947798696e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚865719057117184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202463211 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29929355 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚856645017420800 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚698888193646592 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚876578993073664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88160382 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10859595163136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10840073597620 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚916 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124774400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10840413521054 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89961526 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202473370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114302831 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10859935956480 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60032171 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚687695718962176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0”íkG2Œ - _wandbruntime‚5871  _runtime‚58710n  … /loss/FRt1/assignment_loss‚0.963606378195975 0loss/FRt2/assignment_loss‚1.7038298265703566 3loss/SRqqt2/assignment_loss‚0.46854100952687855 4loss/FBt1/assignment_loss‚1.2867083136845912e-05 2loss/FBt2/assignment_loss‚0.027966284012421738 /loss/FRt1/detection_loss‚1.4255836125070414 0loss/FBt1/detection_loss‚0.39074034171385896 $ _timestamp‚1.784492394780832e+09 trainer/global_step‚11299 /loss/FBt2/detection_loss‚1.3963807462893714 epoch‚14 2loss/SRqqt1/assignment_loss‚0.5114855251798106 _runtime‚5871.355485127 _step‚917 .loss/FRt2/detection_loss‚0.511277701965964 1loss/SRqqt1/detection_loss‚0.6486338441680888 0loss/SRqqt2/detection_loss‚0.882344762299742 %loss/total_loss‚8.930402899512647›ÅŽ Š 3loss/SRqqt2/assignment_loss‚0.46854100952687855 4loss/FBt1/assignment_loss‚1.2867083136845912e-05 2loss/FBt2/assignment_loss‚0.027966284012421738 /loss/FRt1/detection_loss‚1.4255836125070414 0loss/FBt1/detection_loss‚0.39074034171385896 $ _timestamp‚1.784492394780832e+09 trainer/global_step‚11299 /loss/FBt2/detection_loss‚1.3963807462893714 epoch‚14 2loss/SRqqt1/assignment_loss‚0.5114855251798106 _runtime‚5871.355485127 _step‚917 .loss/FRt2/detection_loss‚0.511277701965964 1loss/SRqqt1/detection_loss‚0.6486338441680888 0loss/SRqqt2/detection_loss‚0.882344762299742 %loss/total_loss‚8.930402899512647 /loss/FRt1/assignment_loss‚0.963606378195975 0loss/FRt2/assignment_loss‚1.7038298265703566•'¿Éóe ‚(:Ú óâôÒú©ÂË disk.md0.out‚ 152.65234375" disk.dm-1.out‚9105.43310546875 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375$ memory_percent‚95.48931121826172% disk./.usageGB‚1175.4984512329102# disk.nvme1n1.out‚20505.62109375 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚20505.62109375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625 network.recv‚ 1.577576e+06 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚62342.18017578125% proc.memory.availableMB‚ 739.03125 network.sent‚ 1.5527117e+07 disk.dm-1.in‚0' disk.nvme0c0n1.out‚51270.2900390625% disk.nvme0n1.out‚51270.2900390625" proc.memory.rssMB‚ 2843.9609375( proc.memory.percent‚17.3581600189209 cpu‚26.143313068907073) disk./.usagePercent‚76.56723342992413 disk.nvme1n1.in‚0ll8Ÿ ‚(:” óâôÒ¯­ Ì gpu.0.gpu‚75.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚152.858' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚50.952666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RÀ4·2‘ - _wandbruntime‚5894  _runtime‚5894#4BÇ’ Á _runtime‚5894.949812255 _step‚918 & lr-AdamW/pg1‚0.0001771185235207236 & lr-AdamW/pg2‚0.0001771185235207236 trainer/global_step‚11349 $ _timestamp‚1.784492418374262e+09šõÌ“ Æ _runtime‚5894.949812255 _step‚918 & lr-AdamW/pg1‚0.0001771185235207236 & lr-AdamW/pg2‚0.0001771185235207236 trainer/global_step‚11349 $ _timestamp‚1.784492418374262e+09–ÎYŸh2” - _wandbruntime‚5894  _runtime‚5894.2+ö'• ðO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114798701 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚869354436283904 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 203316536 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114788558 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚880259987662336 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚869354453802496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11239292971520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10905533859840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88527980 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60281140 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚701819952228352 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚869354436283904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚11349 % _timestamp‚1.7844924183749166e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30053201 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88527978 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10905193073664 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚701820107725824 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚880259987662336 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚880259629357568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88527978 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114788558 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚871128483901712 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚860242912501680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚880259629357568 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114798701 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30053492 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90334634 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10905533859840 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 _runtime‚5894.949873646 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚871128840864394 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚860242895462320 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10885588439392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90334341 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 _step‚919 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚690580659256832 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88527980 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10885928362714 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10905193073664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497472 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240®ô óf– õO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚860242895462320 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10885588439392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90334341 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 _step‚919 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚690580659256832 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88527980 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10885928362714 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10905193073664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497472 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114798701 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚869354436283904 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 203316536 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114788558 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚880259987662336 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚869ù]\•354453802496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11239292971520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10905533859840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88527980 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60281140 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚701819952228352 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚869354436283904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚11349 % _timestamp‚1.7844924183749166e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30053201 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88527978 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10905193073664 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚701820107725824 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚880259987662336 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚880259629357568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88527978 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114788558 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚871128483901712 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚860242912501680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚880259629357568 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114798701 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30053492 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90334634 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10905533859840 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 _runtime‚5894.949873646 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚871128840864394—ÀÔÂÛž— ‚(:“ ‚ãôÒõ®êË gpu.0.gpu‚91.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚246.614' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚82.20466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²hYij˜ ‚(:ß ‚ãôÒω´Ì disk.nvme0n1.in‚ 4.3359375' disk.nvme1c1n1.out‚20506.5068359375' proc.memory.availableMB‚ 738.7265625 network.recv‚ 1.581593e+06) disk./.usagePercent‚76.56724436278064% disk.nvme1n1.out‚20506.5068359375& disk.nvme0n1.out‚51274.28662109375 disk.nvme1c1n1.in‚0$ memory_percent‚95.49117088317871" proc.memory.rssMB‚ 2843.9609375 cpu‚26.14239100568224 network.sent‚ 1.5602352e+07 disk.rbd0.out‚ 0.4697265625( disk.nvme0c0n1.out‚51274.28662109375( proc.memory.percent‚17.3581600189209 disk.nvme1n1.in‚0! disk.dm-0.out‚62346.267578125 disk.dm-1.in‚0" disk.dm-1.out‚9105.54248046875 disk.nvme0c0n1.in‚ 4.3359375 proc.cpu.threads‚17% disk./.usageGB‚1175.4986190795898 disk.md0.in‚0 disk.md0.out‚ 152.97265625 disk.dm-0.in‚ 4.3359375 disk.rbd0.in‚0œ¦³2™ - _wandbruntime‚5895  _runtime‚5895ƒÖf-'š úM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚860316321544800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88535480 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _runtime‚5895.308821532 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚880335109349888 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161781760 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10906122984960 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10886516662628 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90341871 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚869428645571584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 203333913 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88535480 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚880334751037952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚690639535589376 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚871202838207428 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10906463778304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114808590 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11240244418560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚869428628052992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90342168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114798433 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114808590 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚297 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10886856586062 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 203333913 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30055945 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚880335109349888 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚701879780007936 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 _step‚920 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚295 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 203344072 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 137357312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚869428645571584 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚690639560013824 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114798433 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚701879941789696 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚871203195170222eë|'› ÿM UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 _step‚920 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚295 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 203344072 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 137357312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚869428645571584 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚690639560013824 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114798433 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚701879941789696 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚871203195170222 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚860316321544800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88535480 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _runtime‚5895.308821532 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚880335109349888 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161781760 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10906122984960 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10886516662628 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90341871 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚869428645571584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 203333913 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88535480 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚880334751037952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚690639535589376 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚871202838207428 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10906463778304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114808590 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11240244418560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚869428628052992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90342168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114798433 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114808590 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚297 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10886856586062 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 203333913 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30055945 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚880335109349888 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚701879780007936˜?Û2œ - _wandbruntime‚5895  _runtime‚5895ˇÿà  … trainer/global_step‚11349 _step‚921 /loss/FRt1/detection_loss‚1.3833605121163761 1loss/SRqqt1/detection_loss‚0.6212530621411222 .loss/FBt2/detection_loss‚0.703403161559254 % _timestamp‚1.7844924187346184e+09 .loss/FRt2/assignment_loss‚1.72330124937851 2loss/SRqqt2/assignment_loss‚0.5676427626320439 %loss/total_loss‚7.997340557741605 epoch‚14 0loss/FRt1/assignment_loss‚0.7831735195617331 2loss/SRqqt1/assignment_loss‚0.4709422352986244 4loss/FBt2/assignment_loss‚3.3638301919580725e-05 1loss/SRqqt2/detection_loss‚0.9151496760942268 0loss/FBt1/detection_loss‚0.34045600759499295 _runtime‚5895.309566813 2loss/FBt1/assignment_loss‚0.006208565588416356 /loss/FRt2/detection_loss‚0.4824161674743858E(ž Š 0loss/FRt1/assignment_loss‚0.7831735195617331 2loss/SRqqt1/assignment_loss‚0.4709422352986244 4loss/FBt2/assignment_loss‚3.3638301919580725e-05 1loss/SRqqt2/detection_loss‚0.9151496760942268 0loss/FBt1/detection_loss‚0.34045600759499295 _runtime‚5895.309566813 2loss/FBt1/assignment_loss‚0.006208565588416356 /loss/FRt2/detection_loss‚0.4824161674743858 trainer/global_step‚11349 _step‚921 /loss/FRt1/detection_loss‚1.3833605121163761 1loss/SRqqt1/detection_loss‚0.6212530621411222 .loss/FBt2/detection_loss‚0.703403161559254 % _timestamp‚1.7844924187346184e+09 .loss/FRt2/assignment_loss‚1.72330124937851 2loss/SRqqt2/assignment_loss‚0.5676427626320439 %loss/total_loss‚7.997340557741605 epoch‚14™3µ][hŸ ‚(:Ý ‘ãôÒòøÝË$ disk./.usageGB‚1175.498634338379 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚20507.34130859375! disk.dm-0.out‚62346.888671875! disk.dm-1.out‚9105.6572265625 disk.nvme0n1.in‚ 4.3359375 disk.nvme1n1.in‚0 disk.md0.out‚ 153.5859375$ memory_percent‚95.48840522766113' proc.memory.availableMB‚ 739.1796875" proc.memory.rssMB‚ 2843.9609375& disk.nvme0n1.out‚51275.46337890625( disk./.usagePercent‚76.5672453566767 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375 disk.md0.in‚0( proc.memory.percent‚17.3581600189209 network.sent‚ 1.5641374e+07& disk.nvme1n1.out‚20507.34130859375 cpu‚26.14144585868267 proc.cpu.threads‚17( disk.nvme0c0n1.out‚51275.46337890625 network.recv‚ 1.585601e+06=’Ûž  ‚(:“ ‘ãôÒæ¸Ì gpu.0.gpu‚85.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚232.942' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚77.64733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ý^Ää2¡ - _wandbruntime‚5918  _runtime‚5918" C2Ê¢ Ä _step‚922 ' lr-AdamW/pg1‚0.00017679957933830172 ' lr-AdamW/pg2‚0.00017679957933830172 trainer/global_step‚11399 % _timestamp‚1.7844924423574748e+09 _runtime‚5918.932438282Ô=¨^Ï£ É ' lr-AdamW/pg1‚0.00017679957933830172 ' lr-AdamW/pg2‚0.00017679957933830172 trainer/global_step‚11399 % _timestamp‚1.7844924423574748e+09 _runtime‚5918.932438282 _step‚922šAÇo;2¤ - _wandbruntime‚5918  _runtime‚5918J‰Ôl¥ ðO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚874846588053402 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60535192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10932371427722 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10952061681664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88903080 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30180448 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚863914199586320 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚884015745633792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 145011712 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚693524475884032 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚884016103938560 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚873064042256896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚873064042256896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90715926 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚874846231090720 % _timestamp‚1.7844924423583317e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88903078 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115284160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚923 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115294303 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10951720895488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10951720895488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11287263271424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚884015745633792 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚704811884167168 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 204197383 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115284160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚923 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115294303 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10951720895488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10951720895488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11287263271424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚884015745633792 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚704811884167168 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 204197383 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88903080 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30180448 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚863914199586320 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚884015745633792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 145011712 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚693524475884032 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚884016103938560 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚873064042256896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚873064042256896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446›å»–c2§ - _wandbruntime‚5919  _runtime‚5919´9p°'¨ úM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60540271 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10933299651070 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60540273 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90723530 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚884091225626112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 204204615 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7844924427178912e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115304192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚874920942359230 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚863987642708160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚884091225626112 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10932959727636 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚873138234025984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚704871574422528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚873138234025984 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10952650806784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88910580 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115294035 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚704871727815680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204204615 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚884090867314176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10952650806784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11288351174656 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚873138216507392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30183257 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90723238 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30182967 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115304192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10952991600128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11288222205952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10952991600128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204214774 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚863987625668800 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88910582 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 _step‚924 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204214774 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚884091225626112 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10932959727636 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚873138234025984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚704871574422528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚873138234025984 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10952650806784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88910580 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115294035 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚704871727815680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204204615 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚884090867314176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10952650806784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11288351174656 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚873138216507392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30183257 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90723238 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30182967 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115304192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10952991600128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11288222205952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10952991600128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204214774 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚863987625668800 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88910582 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 _step‚924 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204214774 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚884091225626112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 204204615 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7844924427178912e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115304192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794œgOÚ2ª - _wandbruntime‚5919  _runtime‚5919êÊ… « „ 1loss/SRqqt2/detection_loss‚0.7648239522826313 /loss/FBt2/detection_loss‚0.8044422401913575 0loss/FRt1/assignment_loss‚0.6838432967135336 /loss/FRt2/assignment_loss‚1.702634908556938 /loss/SRqqt1/assignment_loss‚0.5338033483781 4loss/FBt1/assignment_loss‚0.00037949036782966843 0loss/SRqqt1/detection_loss‚0.569891648504117 % _timestamp‚1.7844924427188716e+09 %loss/total_loss‚8.148859782489467 trainer/global_step‚11399 _step‚925 /loss/FBt1/detection_loss‚0.4478475135270386 2loss/SRqqt2/assignment_loss‚0.6201819731740529 4loss/FBt2/assignment_loss‚0.00017931667746446628 /loss/FRt2/detection_loss‚0.5995617847976682 /loss/FRt1/detection_loss‚1.4212703093187369 epoch‚14 _runtime‚5919.293512143Í©"Œ¬ ‰ 1loss/SRqqt2/detection_loss‚0.7648239522826313 /loss/FBt2/detection_loss‚0.8044422401913575 0loss/FRt1/assignment_loss‚0.6838432967135336 /loss/FRt2/assignment_loss‚1.702634908556938 /loss/SRqqt1/assignment_loss‚0.5338033483781 4loss/FBt1/assignment_loss‚0.00037949036782966843 0loss/SRqqt1/detection_loss‚0.569891648504117 % _timestamp‚1.7844924427188716e+09 %loss/total_loss‚8.148859782489467 trainer/global_step‚11399 _step‚925 /loss/FBt1/detection_loss‚0.4478475135270386 2loss/SRqqt2/assignment_loss‚0.6201819731740529 4loss/FBt2/assignment_loss‚0.00017931667746446628 /loss/FRt2/detection_loss‚0.5995617847976682 /loss/FRt1/detection_loss‚1.4212703093187369 epoch‚14 _runtime‚5919.293512143(<üìk­ ‚(:à  ãôÒ¢‘‡Ì( disk./.usagePercent‚76.5672627498575 disk.nvme0n1.in‚ 4.3359375% disk.nvme0n1.out‚51280.3076171875 disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625' disk.nvme0c0n1.out‚51280.3076171875$ memory_percent‚95.48876285552979( proc.memory.percent‚17.3581600189209 network.sent‚ 1.570649e+07 disk.nvme1c1n1.in‚0! disk.dm-1.out‚9105.7939453125 proc.cpu.threads‚17( disk.nvme1c1n1.out‚20508.16748046875 disk.nvme1n1.in‚0& disk.nvme1n1.out‚20508.16748046875( proc.memory.availableMB‚ 739.12109375" proc.memory.rssMB‚ 2843.9609375 cpu‚26.140496287631247 network.recv‚ 1.589064e+06% disk./.usageGB‚1175.4989013671875 disk.md0.in‚0 disk.md0.out‚ 153.91796875 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚62351.69970703125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375ÍÐÌ‘® ‚(:†  ãôÒØŒ»Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚247.755' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚82.585 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½ýXèe¯ ‚(:Ú ¯ãôÒøÇÌ network.recv‚ 1.592958e+06& disk.nvme0c0n1.out‚51281.560546875& disk.nvme1c1n1.out‚20508.744140625) disk./.usagePercent‚76.39027495975118# disk.dm-0.out‚62352.65673828125 disk.dm-1.in‚0" disk.dm-1.out‚9105.88720703125 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375$ disk.nvme1n1.out‚20508.744140625 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0$ memory_percent‚95.48876285552979( proc.memory.availableMB‚ 739.12109375" proc.memory.rssMB‚ 2843.9609375( proc.memory.percent‚17.3581600189209 network.sent‚ 1.5756947e+07# disk./.usageGB‚1172.78169631958 disk.dm-0.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375$ disk.nvme0n1.out‚51281.560546875 disk.md0.out‚ 154.27734375 cpu‚26.13959180705914 proc.cpu.threads‚17ut[$ž° ‚(:“ ¯ãôÒ¢Ê‚Í gpu.0.gpu‚96.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚262.204' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.40133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à¸ï;2± - _wandbruntime‚5942  _runtime‚5942s<§Ãɲ à ' lr-AdamW/pg1‚0.00017647950795821372 ' lr-AdamW/pg2‚0.00017647950795821372 trainer/global_step‚11449 % _timestamp‚1.7844924660099664e+09 _runtime‚ 5942.58494565 _step‚926­ €Î³ È ' lr-AdamW/pg1‚0.00017647950795821372 ' lr-AdamW/pg2‚0.00017647950795821372 trainer/global_step‚11449 % _timestamp‚1.7844924660099664e+09 _runtime‚ 5942.58494565 _step‚926ž|!Dä2´ - _wandbruntime‚5942  _runtime‚5942B%µö'µ ðO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115779762 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91096607 % _timestamp‚1.7844924660108814e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10998589503488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89278180 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60789240 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚878564335242410 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚867585520749680 trainer/global_step‚11449 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚876773630711296 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚876773613192704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115789905 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10998248717312 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115789905 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89278178 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚867585503710320 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚696468292511232 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚887771861910016 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚887771861910016 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚887771861910016 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115779762 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91096607 % _timestamp‚1.7844924660108814e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10998589503488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89278180 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60789240 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚878564335242410 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚867585520749680 trainer/global_step‚11449 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚876773630711296 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚876773613192704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115789905 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10998248717312 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115789905 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89278178 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚867585503710320 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚696468292511232 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚887771861910016ŸØ]½¼2· - _wandbruntime‚5943  _runtime‚5943 N}-I¸ ùM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10999178628608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60794323 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30309942 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115799794 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30309649 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155490304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚_&ý¸¶" 13862174720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91103970 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89285680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11336215984128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10999178628608 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11336347049984 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚867658929792800 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚876847804961792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91104265 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚696527168843776 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10999519421952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚295 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 trainer/global_step‚11449 $ _timestamp‚1.784492466685677e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205085476 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115789637 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚707863384827904 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89285682 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10999519421952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115799794 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚707863540318208 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚876847822480384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205075317 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚867658946832160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚696527193268224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚887847341902336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚887847341902336 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 _runtime‚5943.260844781 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚887846983590400 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10979742716078 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚876847822480384 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115789637 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60794321 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205085476 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚707863540318208 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚876847822480384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205075317 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚867658946832160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚696527193268224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚887847341902336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚887847341902336 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 _runtime‚5943.260844781 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚887846983590400 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10979742716078 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚876847822480384 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115789637 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60794321 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205085476 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91103970 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89285680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11336215984128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10999178628608 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11336347049984 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚867658929792800 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚876847804961792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91104265 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚696527168843776 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10999519421952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚295 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 trainer/global_step‚11449 $ _timestamp‚1.784492466685677e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205085476 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115789637 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚707863384827904 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89285682 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10999519421952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115799794 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 Z©ñÂ2º - _wandbruntime‚5943  _runtime‚5943-(ýŒ» ‚ 2loss/SRqqt2/assignment_loss‚0.4971375471565467 2loss/FBt1/assignment_loss‚0.021631756819229048 2loss/FBt2/assignment_loss‚0.001739288777978724 /loss/FRt2/detection_loss‚0.5024289263830287 /loss/FBt1/detection_loss‚0.6434037436724601 0loss/FRt1/assignment_loss‚0.8823630866550264 1loss/SRqqt2/detection_loss‚0.9189734596650522 -loss/FBt2/detection_loss‚0.92450754220287 trainer/global_step‚11449 _runtime‚5943.261949793 0loss/FRt2/assignment_loss‚1.7883134597539903 %loss/total_loss‚8.689924591220425 epoch‚14 $ _timestamp‚1.784492466687185e+09 2loss/SRqqt1/assignment_loss‚0.5093613931952996 /loss/FRt1/detection_loss‚1.3394530130463584 1loss/SRqqt1/detection_loss‚0.6606113738925841 _step‚929¤ª²j ¼ ‡ 0loss/FRt1/assignment_loss‚0.8823630866550264 1loss/SRqqt2/detection_loss‚0.9189734596650522 -loss/FBt2/detection_loss‚0.92450754220287 trainer/global_step‚11449 _runtime‚5943.261949793 0loss/FRt2/assignment_loss‚1.7883134597539903 %loss/total_loss‚8.689924591220425 epoch‚14 $ _timestamp‚1.784492466687185e+09 2loss/SRqqt1/assignment_loss‚0.5093613931952996 /loss/FRt1/detection_loss‚1.3394530130463584 1loss/SRqqt1/detection_loss‚0.6606113738925841 _step‚929 2loss/SRqqt2/assignment_loss‚0.4971375471565467 2loss/FBt1/assignment_loss‚0.021631756819229048 2loss/FBt2/assignment_loss‚0.001739288777978724 /loss/FRt2/detection_loss‚0.5024289263830287 /loss/FBt1/detection_loss‚0.6434037436724601¡¬¡!0e½ ‚(:Ú ¾ãôÒº¤¦Ì disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 4.3359375$ memory_percent‚95.48892974853516% proc.memory.availableMB‚ 739.09375 network.recv‚ 1.596462e+06) disk./.usagePercent‚76.39026874790089" disk.dm-0.out‚62356.2119140625 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625& disk.nvme0n1.out‚51285.10791015625& disk.nvme1c1n1.out‚20512.580078125" proc.memory.rssMB‚ 2843.9609375" disk.dm-1.out‚9109.15283203125 disk.nvme1c1n1.in‚0 cpu‚26.138573285574186 proc.cpu.threads‚17 network.sent‚ 1.5772754e+07% disk./.usageGB‚1172.7816009521484 disk.md0.out‚ 154.53125( disk.nvme0c0n1.out‚51285.10791015625 disk.nvme0n1.in‚ 4.3359375 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚20512.580078125( proc.memory.percent‚17.3581600189209 disk.md0.in‚0õ1!bž¾ ‚(:“ ¾ãôÒ¥¥ÞÌ gpu.0.gpu‚70.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚255.895' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.29833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©wH2¿ - _wandbruntime‚5966  _runtime‚5966º]ìŒÉÀ à ' lr-AdamW/pg1‚0.00017615831450705242 ' lr-AdamW/pg2‚0.00017615831450705242 trainer/global_step‚11499 $ _timestamp‚1.784492490007945e+09 _runtime‚5966.582880287 _step‚930uNOŸÎÁ È ' lr-AdamW/pg1‚0.00017615831450705242 ' lr-AdamW/pg2‚0.00017615831450705242 trainer/global_step‚11499 $ _timestamp‚1.784492490007945e+09 _runtime‚5966.582880287 _step‚930¢ÚšÙ“2 - _wandbruntime‚5966  _runtime‚5966R³©Íõ'à ïO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89653280 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _runtime‚5966.583688138 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚880483219165696 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11045117325312 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11044776539136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89653280 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89653278 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89653278 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116285507 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚699412133562880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚871256807834320 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205928642 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11383230286336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚882281725468736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11024917634416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30433646 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11383365553664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116275364 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚880483219165696 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚880483201647104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61043290 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91476938 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159691776 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚699412109138432 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚891528336491008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30433353 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11044776539136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 135267328 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚891527978186240 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11045117325312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚880483201647104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚710795339424768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768óÛ£f±Ä ôO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on<àOSI'_train_batch_start/active_bytes.all.current‚ 358304768 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89653280 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _runtime‚5966.583688138 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚880483219165696 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11045117325312 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11044776539136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89653280 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89653278 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89653278 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116285507 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚699412133562880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚871256807834320 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205928642 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11383230286336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚882281725468736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11024917634416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30433646 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11383365553664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116275364 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚880483219165696 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚880483201647104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61043290 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91476938 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159691776 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚699412109138432 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚891528336491008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30433353 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11044776539136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 135267328 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚891527978186240 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11045117325312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚880483201647104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚710795339424768£![³‰2Å - _wandbruntime‚5966  _runtime‚5966«X¤j'Æ úM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚880557410934784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚699470985470976 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚891603458178560 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89660782 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116295396 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚880557393416192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 _step‚932 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚891603099866624 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11384192061440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116295396 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61048371 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205956178 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚710855177532416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11384321030144 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚882356079774452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89660780 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 % _timestamp‚1.7844924903662324e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚880557410934784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91484176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚699471009895424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚871330250956160 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 trainer/global_step‚11499 _runtime‚5966.941361838 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚882356436737246 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205946019 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚871330233916800 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚891603458178560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚299 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11046047243776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11025845857652 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520_+³î'Ç ÿM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚891603458178560 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89660782 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116295396 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚880557393416192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 _step‚932 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚891603099866624 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11384192061440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116295396 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61048371 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205956178 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚710855177532416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11384321030144 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚882356079774452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89660780 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 % _timestamp‚1.7844924903662324e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚880557410934784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91484176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚699471009895424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153393152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚871330250956160 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 trainer/global_step‚11499 _runtime‚5966.941361838 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚882356436737246 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205946019 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚871330233916800 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚891603458178560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚299 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11046047243776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11025845857652 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚880557410934784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚699470985470976¤£¯ÿ÷2È - _wandbruntime‚5966  _runtime‚5966úÞ·Ÿ É „ /loss/FRt1/assignment_loss‚0.763922994179576 /loss/FRt1/detection_loss‚1.3951068372484463 /loss/FRt2/detection_loss‚0.5401199321244899 1loss/SRqqt2/detection_loss‚0.9236047543486445 % _timestamp‚1.7844924903678448e+09 2loss/SRqqt1/assignment_loss‚0.5297215268181555 2loss/SRqqt2/assignment_loss‚0.5475182156424938 2loss/FBt1/assignment_loss‚0.002470320499786263 0loss/FBt1/detection_loss‚0.38326815543566933 %loss/total_loss‚8.531938171663473 epoch‚14 _runtime‚ 5966.94249328 4loss/FBt2/assignment_loss‚0.00031025692909939347 -loss/FBt2/detection_loss‚1.12055547262675 _step‚933 0loss/FRt2/assignment_loss‚1.7693434279374403 1loss/SRqqt1/detection_loss‚0.5559962778729228 trainer/global_step‚11499“$Ù†Ê ‰ trainer/global_step‚11499 /loss/FRt1/assignment_loss‚0.763922994179576 /loss/FRt1/detection_loss‚1.3951068372484463 /loss/FRt2/detection_loss‚0.5401199321244899 1loss/SRqqt2/detection_loss‚0.9236047543486445 % _timestamp‚1.7844924903678448e+09 2loss/SRqqt1/assignment_loss‚0.5297215268181555 2loss/SRqqt2/assignment_loss‚0.5475182156424938 2loss/FBt1/assignment_loss‚0.002470320499786263 0loss/FBt1/detection_loss‚0.38326815543566933 %loss/total_loss‚8.531938171663473 epoch‚14 _runtime‚ 5966.94249328 4loss/FBt2/assignment_loss‚0.00031025692909939347 -loss/FBt2/detection_loss‚1.12055547262675 _step‚933 0loss/FRt2/assignment_loss‚1.7693434279374403 1loss/SRqqt1/detection_loss‚0.5559962778729228¥>솮kË ‚(:à ÍãôÒõùÄÌ$ memory_percent‚95.48919200897217 proc.cpu.threads‚17 disk.nvme0n1.in‚ 4.3359375 network.recv‚ 1.600545e+06( disk./.usagePercent‚76.3903070128987 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1n1.in‚0% disk.nvme1n1.out‚20513.1943359375 disk.md0.in‚0 network.sent‚ 1.5822363e+07 disk.dm-0.in‚ 4.3359375" disk.dm-0.out‚62357.0400390625 disk.rbd0.in‚0 disk.md0.out‚ 154.8984375( proc.memory.availableMB‚ 739.05078125" proc.memory.rssMB‚ 2843.9609375( proc.memory.percent‚17.3581600189209% disk./.usageGB‚1172.7821884155273" disk.dm-1.out‚9109.28955078125 disk.rbd0.out‚ 0.4697265625( disk.nvme0c0n1.out‚51286.27294921875 disk.nvme1c1n1.in‚0 cpu‚26.137764338320206& disk.nvme0n1.out‚51286.27294921875' disk.nvme1c1n1.out‚20513.1943359375>fðÆ‹Ì ‚(:‘ ÍãôÒ½ªóË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedByteTšs‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚263.147' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.71566666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀTêbÍ ‚(:× ÜãôÒõéË disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 155.2734375 disk.dm-0.in‚ 4.3359375 disk.nvme1c1n1.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375& disk.nvme0n1.out‚51295.57177734375! disk.dm-0.out‚62367.599609375" disk.dm-1.out‚9109.43408203125 disk.rbd0.out‚ 0.4697265625 network.sent‚ 1.5886952e+07) disk./.usagePercent‚76.39030452815858" disk.nvme1n1.out‚ 20515.4140625( disk.nvme0c0n1.out‚51295.57177734375$ disk.nvme1c1n1.out‚ 20515.4140625% disk./.usageGB‚1172.7821502685547 disk.dm-1.in‚0 disk.nvme0n1.in‚ 4.3359375$ memory_percent‚95.48940658569336& proc.memory.availableMB‚ 739.015625" proc.memory.rssMB‚ 2843.9609375( proc.memory.percent‚17.3581600189209 cpu‚26.13863478084859 network.recv‚ 1.605139e+06÷gãžÎ ‚(:“ ÜãôÒàýšÌ gpu.0.gpu‚58.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚161.451' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚53.81699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùÒ˜o2Ï - _wandbruntime‚5992  _runtime‚5992XùBtÊÐ Ä ' lr-AdamW/pg1‚0.00017583600412938289 ' lr-AdamW/pg2‚0.00017583600412938289 trainer/global_step‚11549 % _timestamp‚1.7844925161549962e+09 _runtime‚5992.730025523 _step‚934M^ÏÑ É ' lr-AdamW/pg1‚0.00017583600412938289 ' lr-AdamW/pg2‚0.00017583600412938289 trainer/global_step‚11549 % _timestamp‚1.7844925161549962e+09 _runtime‚5992.730025523 _step‚934¦úk‰2Ò - _wandbruntime‚5992  _runtime‚5992GÅ$ö'Ó ðO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61297342 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30560199 % _timestamp‚1.7844925161560817e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚702355925765632 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11091304360960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90028380 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116770966 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11091645147136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚702355950190080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚5992.731141584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91857841 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61297340 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚874928111958320 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116781109 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116770966 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚91857539 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11091304360960 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90028378 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚895284094462464 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 206809489 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚895284452767232 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚895284094462464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚874928128997680 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚713787107825664 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚884192807620096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90028380 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90028378 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚884192790101504 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11431311035904 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 206799344 trainer/global_step‚11549 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚884192790101504 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _step‚935 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚884192807620096 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚302 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚244244481+û'Ô õO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61297340 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚874928111958320 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116781109 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116770966 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚91857539 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11091304360960 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90028378 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚895284094462464 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 206809489 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚895284452767232 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚895284094462464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚874928128997680 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚713787107825664 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚884192807620096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90028380 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90028378 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚884192790101504 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11431311035904 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 206799344 trainer/global_step‚11549 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚884192790101504 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _step‚935 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚884192807620096 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚302 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61297342 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30560199 % _timestamp‚1.7844925161560817e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚702355925765632 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11091304360960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90028380 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116770966 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11091645147136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚702355950190080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚5992.731141584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91857841 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚884266999389184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚895359216142848 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11092234272256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚884266981870592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11092575065600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30563016 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11092575065600 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚702414826522624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚886073826963460 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚875001555080160 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116780841 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90035882 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91865150 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 206826880 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 206826880 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30562729 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚886074183926254 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚713846949793792 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 206816721 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚895359574454784 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11432268275712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11432147695616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11072288922660 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90035880 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚702414802098176 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚875001538040800 _step‚936 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116790998 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚884266981870592 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11092234272256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚884266999389184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90035882 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116780841 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 trainer/global_step‚11549XO×|× þM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 MCDeviceStatsMonitor.on_trainJ®„ãˆ&_batch_end/allocation.small_pool.current‚10157 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 206826880 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 206826880 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30562729 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚886074183926254 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚713846949793792 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 206816721 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚895359574454784 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11432268275712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11432147695616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11072288922660 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90035880 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚702414802098176 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚875001538040800 _step‚936 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116790998 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚884266981870592 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11092234272256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚884266999389184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90035882 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116780841 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 trainer/global_step‚11549 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 145004544 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚884266999389184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚895359216142848 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11092234272256 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚884266981870592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11092575065600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30563016 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11092575065600 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚702414826522624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚886073826963460 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚875001555080160 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116780841 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90035882 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91865150¨¯Ä-ÿ2Ø - _wandbruntime‚5993  _runtime‚5993 Š»ó Ù … 2loss/SRqqt2/assignment_loss‚0.7398078520260573 2loss/FBt1/assignment_loss‚0.007176653921231472 /loss/FRt1/detection_loss‚1.2827560342766964 1loss/SRqqt1/detection_loss‚0.6712364252843247 1loss/SRqqt2/detection_loss‚0.8554600909799821 /loss/FBt1/detection_loss‚0.4335070933486956 0loss/FRt1/assignment_loss‚0.8382517099380493 /loss/FRt2/assignment_loss‚1.726743959578184 4loss/FBt2/assignment_loss‚0.00044066317316650484 .loss/FBt2/detection_loss‚2.264372787039195 /loss/FRt2/detection_loss‚0.5318476925195752 %loss/total_loss‚9.863016503373533 epoch‚14 _step‚937 2loss/SRqqt1/assignment_loss‚0.5114155412883762 trainer/global_step‚11549 % _timestamp‚1.7844925165405412e+09 _runtime‚5993.115259161­ÅQ…Ú Š 1loss/SRqqt1/detection_loss‚0.6712364252843247 1loss/SRqqt2/detection_loss‚0.8554600909799821 /loss/FBt1/detection_loss‚0.4335070933486956 0loss/FRt1/assignment_loss‚0.8382517099380493 /loss/FRt2/assignment_loss‚1.726743959578184 4loss/FBt2/assignment_loss‚0.00044066317316650484 .loss/FBt2/detection_loss‚2.264372787039195 /loss/FRt2/detection_loss‚0.5318476925195752 %loss/total_loss‚9.863016503373533 epoch‚14 _step‚937 2loss/SRqqt1/assignment_loss‚0.5114155412883762 trainer/global_step‚11549 % _timestamp‚1.7844925165405412e+09 _runtime‚5993.115259161 2loss/SRqqt2/assignment_loss‚0.7398078520260573 2loss/FBt1/assignment_loss‚0.007176653921231472 /loss/FRt1/detection_loss‚1.2827560342766964©à®»ºlÛ ‚(:á ëãôÒÂÈË% disk./.usageGB‚1172.7821655273438( disk.nvme1c1n1.out‚20515.98193359375 disk.md0.out‚ 155.52734375( proc.memory.percent‚17.3581600189209 proc.cpu.threads‚17) disk./.usagePercent‚76.39030552205463! disk.dm-1.out‚9109.6552734375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚51296.31298828125 disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚51296.31298828125 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚20515.98193359375 network.sent‚ 1.5889874e+07 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375( proc.memory.availableMB‚ 738.73046875" proc.memory.rssMB‚ 2843.9609375 network.recv‚ 1.608629e+06 disk.dm-1.in‚0! disk.dm-0.out‚62368.119140625# memory_percent‚95.4911470413208 cpu‚26.140162030839733V6ížÜ ‚(:“ ëãôÒ‡Ø·Ë gpu.0.gpu‚89.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚119.284' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.76133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jgE‚iÝ ‚(:Þ úãôÒßÒ÷Ë" proc.memory.rssMB‚ 2843.9609375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375' disk.nvme1c1n1.out‚20516.6806640625 disk.md0.in‚0 cpu‚26.141592993964345 proc.cpu.threads‚17 network.sent‚ 1.595437e+07) disk./.usagePercent‚76.38980708318704 disk.rbd0.out‚ 0.4697265625& disk.nvme0n1.out‚51298.32958984375 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 155.84765625# disk.dm-0.out‚62370.06591796875 network.recv‚ 1.612683e+06( disk.nvme0c0n1.out‚51298.32958984375 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0! disk.dm-1.out‚9109.7294921875$ memory_percent‚95.48959732055664& proc.memory.availableMB‚ 738.984375( proc.memory.percent‚17.3581600189209$ disk./.usageGB‚1172.774513244629% disk.nvme1n1.out‚20516.6806640625·ûœ‰žÞ ‚(:“ úãôÒº•¨Ì gpu.0.gpu‚51.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚255.898' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.29933333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F+&h2ß - _wandbruntime‚6018  _runtime‚6018‚žÐpÈà  trainer/global_step‚11599 # _timestamp‚1.78449254231395e+09 _runtime‚6018.888936682 _step‚938 ' lr-AdamW/pg1‚0.00017551258198766006 ' lr-AdamW/pg2‚0.00017551258198766006þ’“Íá Ç trainer/global_step‚11599 # _timestamp‚1.78449254231395e+09 _runtime‚6018.888936682 _step‚938 ' lr-AdamW/pg1‚0.00017551258198766006 ' lr-AdamW/pg2‚0.00017551258198766006ª½Ûd2â - _wandbruntime‚6018  _runtime‚6018¸\ûÌö'ã ðO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 _step‚939 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11138172968960 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11137832182784 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11118143687754 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61551390 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207680191 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚11599 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚899040210738688 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117276711 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚887902378555904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚705299742392832 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30687499 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11479193790464 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚705299766817280 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚899040569043456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚716778936183296 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚899040210738688 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92238891 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92238598 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚887902396074496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11137832182784 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117266568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚887902378555904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90403478 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61551392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚899040569043456 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11138172968960 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11479320669184 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11117803764432 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚887902396074496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚716779087486464 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚889717576809434 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚878599433121680 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90403480 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 % _timestamp‚1.7844925423149457e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚889717219846752F¨ÔÉä õO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚887902378555904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚705299742392832 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30687499 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11479193790464 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚705299766817280 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚899040569043456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚716778936183296 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚899040210738688 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92238891 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92238598 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚887902396074496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11137832182784 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117266568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚887902378555904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90403478 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61551392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚899040569043456 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11138172968960 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11479320669184 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11117803764432 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚887902396074496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚716779087486464 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚889717576809434 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚878599433121680 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90403480 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 % _timestamp‚1.7844925423149457e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚889717219846752 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151303168 _step‚939 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11138172968960 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11137832182784 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11118143687754 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61551390 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207680191 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚11599 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚899040210738688 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117276711 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126878720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0«)wÑl2å - _wandbruntime‚6019  _runtime‚6019‰*ÿ&æ ùM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚889791931115262 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207697582 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11480283520512 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚887976570324992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30689736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30690030 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11480154551808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117276443 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92246207 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11139102887424 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚294 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚887976587843584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61556471 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚887976587843584 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207697582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11139102887424 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚899115690731008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92246503 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117276443 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚6019.264510141 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207687423 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚899115690731008 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚705358618725376^Åê†'ç þM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚887976570324992 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚716838773277184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚899115690731008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92246503 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128968704 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117276443 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚6019.264510141 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207687423 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚899115690731008 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚705358618725376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚889791931115262 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207697582 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11480283520512 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚887976570324992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30689736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30690030 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11480154551808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117276443 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92246207 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11139102887424 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚294 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚887976587843584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61556471 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚887976587843584 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207697582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11139102887424 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92618984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11184360004608 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚891611984528896 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 % _timestamp‚1.7844925684201264e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11184700790784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 208550893 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90778580 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92619275 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61805442 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚902796327014912 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚708243559020032 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11527333990400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11164586752762 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11184360004608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚902796327014912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚891611984528896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11164246829440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 trainer/global_step‚11649 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90778580 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90778578 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚891611967010304 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚708243583444480 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11184700790784 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚882270737245680 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11527211305984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90778578 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117762170 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61805440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147108864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚882270720206320 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30813833 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚719770917434880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_8·ëÛý batch_start/inactive_split.all.peak‚837 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122684416 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11184700790784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 208550893 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90778580 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92619275 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61805442 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚902796327014912 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚708243559020032 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11527333990400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11164586752762 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11184360004608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚902796327014912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚891611984528896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11164246829440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 trainer/global_step‚11649 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90778580 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90778578 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚891611967010304 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚708243583444480 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11184700790784 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚882270737245680 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11527211305984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90778578 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117762170 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61805440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147108864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚882270720206320 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30813833 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚719770917434880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122684416 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92618984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11184360004608 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚891611984528896 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 % _timestamp‚1.7844925684201264e+09¯ ž(Phó ‚(:Ý ˜äôÒŸñÇË network.sent‚ 1.6020969e+07 disk.nvme1c1n1.in‚0 disk.md0.out‚ 156.60546875# disk.dm-0.out‚62372.98486328125 disk.rbd0.in‚0" proc.memory.rssMB‚ 2843.9609375 proc.cpu.threads‚17) disk./.usagePercent‚76.38985752341142 disk.nvme0n1.in‚ 4.3359375# disk.nvme0n1.out‚51301.80859375& disk.nvme1n1.out‚20517.95361328125 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375$ memory_percent‚95.49129009246826( proc.memory.percent‚17.3581600189209 network.recv‚ 1.62034e+06% disk./.usageGB‚1172.7752876281738( disk.nvme1c1n1.out‚20517.95361328125 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375! disk.dm-1.out‚9109.9267578125 cpu‚26.144608837106013 disk.nvme1n1.in‚0 disk.dm-1.in‚0% disk.nvme0c0n1.out‚51301.80859375( proc.memory.availableMB‚ 738.70703125RÅ ÿô ‚(:’ ˜äôÒú ¦Ì gpu.0.gpu‚12.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚52.0 gpu.0.powerWatts‚118.264' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.42133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô©ºÞ2õ - _wandbruntime‚6045  _runtime‚6045e/'ö úM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 118482944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208568284 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90786080 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117772045 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚893509321341476 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30816026 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 208558125 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚902871807007232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 % _timestamp‚1.7844925687978282e+09 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚708302435352576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚902871448695296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 208558125 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92626830 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11165514976110 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11165175052676 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚902871807007232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 _runtime‚6045.372991218 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚893509678304270 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚891686176297984 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚882344146288800 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 208568284 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117782202 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90786082 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90786080 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11185289915904 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92626547 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61810521 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚891686158779392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 _step‚944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚891686176297984 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117782202 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 142907392 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11528291463680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0·Hÿš ÷ ÿM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11185289915904 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92626547 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61810521 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚891686158779392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 _step‚944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚891686176297984 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117782202 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 142907392 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11528291463680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 118482944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208568284 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90786080 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117772045 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚893509321341476 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30816026 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 208558125 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚902871807007232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 % _timestamp‚1.7844925687978282e+09 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚708302435352576 GADeviceStatsMonitor.on_traiÀ•ÐŒen_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚902871448695296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 208558125 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92626830 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11165514976110 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11165175052676 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚902871807007232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 _runtime‚6045.372991218 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚893509678304270 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚891686176297984 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚882344146288800 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 208568284 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117782202 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90786082 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90786080°v]ö 2ø - _wandbruntime‚6045  _runtime‚6045û ù … epoch‚14 trainer/global_step‚11649 4loss/FBt2/assignment_loss‚0.00011619892736714586 0loss/FBt1/detection_loss‚0.43873962042638787 /loss/FBt2/detection_loss‚0.7283399249700939 %loss/total_loss‚8.359192586663909 _runtime‚ 6045.37381651 /loss/FRt2/assignment_loss‚1.681394221760192 % _timestamp‚1.7844925687990718e+09 2loss/SRqqt1/assignment_loss‚0.4948738552046102 2loss/SRqqt2/assignment_loss‚0.6372389891245418 _step‚945 0loss/FRt1/assignment_loss‚0.6388769268437668 2loss/FBt1/assignment_loss‚0.003749672321489166 .loss/FRt1/detection_loss‚1.429055359647868 /loss/FRt2/detection_loss‚0.5805279436388892 1loss/SRqqt1/detection_loss‚0.6791671392229021 1loss/SRqqt2/detection_loss‚1.0471127345757996Vkgú Š 4loss/FBt2/assignment_loss‚0.00011619892736714586 0loss/FBt1/detection_loss‚0.43873962042638787 /loss/FBt2/detection_loss‚0.7283399249700939 %loss/total_loss‚8.359192586663909 _runtime‚ 6045.37381651 /loss/FRt2/assignment_loss‚1.681394221760192 % _timestamp‚1.7844925687990718e+09 2loss/SRqqt1/assignment_loss‚0.4948738552046102 2loss/SRqqt2/assignment_loss‚0.6372389891245418 _step‚945 0loss/FRt1/assignment_loss‚0.6388769268437668 2loss/FBt1/assignment_loss‚0.003749672321489166 .loss/FRt1/detection_loss‚1.429055359647868 /loss/FRt2/detection_loss‚0.5805279436388892 1loss/SRqqt1/detection_loss‚0.6791671392229021 1loss/SRqqt2/detection_loss‚1.0471127345757996 epoch‚14 trainer/global_step‚11649±A:QNeû ‚(:Ú §äôÒÒéÅË disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚51306.30029296875 disk.nvme1n1.in‚0 disk.dm-1.out‚ 9110.0703125 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625# memory_percent‚95.4901933670044 network.sent‚ 1.60727e+07 network.recv‚ 1.624813e+06( disk.nvme0c0n1.out‚51306.30029296875 disk.nvme1c1n1.in‚0# disk.dm-0.out‚62377.32470703125( proc.memory.availableMB‚ 738.88671875" proc.memory.rssMB‚ 2843.9609375 proc.cpu.threads‚17( proc.memory.percent‚17.3581600189209 cpu‚26.14614521139621) disk./.usagePercent‚76.39320347445386% disk./.usageGB‚1172.8266563415527' disk.nvme1c1n1.out‚20518.4130859375% disk.nvme1n1.out‚20518.4130859375 disk.md0.in‚0 disk.md0.out‚ 156.8203125 disk.dm-0.in‚ 4.3359375 disk.dm-1.in‚0•^žü ‚(:“ §äôÒÓ»¸Ì gpu.0.gpu‚55.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚267.113' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.03766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z1Žt2ý - _wandbruntime‚6070  _runtime‚6070לV;Êþ Ä _step‚946 ' lr-AdamW/pg1‚0.00017486242315082684 ' lr-AdamW/pg2‚0.00017486242315082684 trainer/global_step‚11699 % _timestamp‚1.7844925941809807e+09 _runtime‚6070.755985778/d„Ïÿ É ' lr-AdamW/pg2‚0.00017486242315082684 trainer/global_step‚11699 % _timestamp‚1.7844925941809807e+09 _runtime‚6070.755985778 _step‚946 ' lr-AdamW/pg1‚0.00017486242315082684²õ0®J2€!- _wandbruntime‚6070  _runtime‚6070)¶cô'!îO P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 _runtime‚ 6070.75703676 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11230887826432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91153680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30940324 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11211029817770 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 _step‚947 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚305 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚722762540118016 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚711187375647232 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚906552801595904 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118257772 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11575295543808 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11210689894448 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚906552801595904 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118267915 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 209411450 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91153680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497472 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚895321572983296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚897152714224768 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚885942041369680 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11575164470784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚906552443291136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91153678 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91153678 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93000119 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209421595 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚711187400071680 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚906552443291136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 $ _timestamp‚1.784492594182075e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11231228612608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30940627 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚897153071187450 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118267915 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚895321555464704 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11231228612608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11230887826432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62059492 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚722762695615488 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0ÍQù'‚!óO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 209411450 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91153680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155497472 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚895321572983296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚897152714224768 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚885942041369680 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11575164470784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚906552443291136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91153678 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91153678 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93000119 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209421595 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚711187400071680 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚906552443291136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 $ _timestamp‚1.784492594182075e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11231228612608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30940627 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚897153071187450 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118267915 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚895321555464704 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11231228612608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11230887826432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62059492 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚722762695615488 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 _runtime‚ 6070.75703676 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11230887826432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91153680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30940324 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11211029817770 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 _step‚947 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚305 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚722762540118016 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚711187375647232 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚906552801595904 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118257772 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11575295543808 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11210689894448 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚906552801595904 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118267915³QLW2ƒ!- _wandbruntime‚6071  _runtime‚6071Wփ̈́!úM EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚906627564971520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚711246251979776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91161180 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91161182 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118267647 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93007382 % _timestamp‚1.7844925945570045e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30943101 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 _step‚948 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62064573 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11211618117684 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118267647 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91161182 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30942811 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚895395747233792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 trainer/global_step‚11699 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚722822530830336 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚895395764752384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 1 3 14407434240 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚897227425493278 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209438986 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚897227068530484 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚906627923283456 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11232158531072 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11231817737728 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚722822375340032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚895395764752384 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0˜Å'…!ÿM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118267647 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93007382 % _timestamp‚1.7844925945570045e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30943101 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 _step‚948 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62064573 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11211618117684 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118267647 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 131065856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91161182 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30942811 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚895395747233792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 trainer/global_step‚11699 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚722822530830336 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚895395764752384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚897227425493278 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209438986 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚897227068530484 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚906627923283456 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11232158531072 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11231817737728 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚722822375340032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚895395764752384 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚906627564971520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚711246251979776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91161180 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91161182´ÍÕüÍ2†!- _wandbruntime‚6071  _runtime‚6071ŸÐ… ‡!„ % _timestamp‚1.7844925945580835e+09 _step‚949 0loss/FRt2/assignment_loss‚1.6095139884400642 /loss/FRt1/detection_loss‚1.3869279400067038 /loss/FBt2/detection_loss‚0.8677486477662688 _runtime‚6071.132760783 1loss/SRqqt2/detection_loss‚0.8025897804196134 /loss/FBt1/detection_loss‚0.4486955418517547 trainer/global_step‚11699 2loss/SRqqt1/assignment_loss‚0.5357670756215805 2loss/SRqqt2/assignment_loss‚0.5498417903210573 1loss/FBt1/assignment_loss‚0.03644859881081501 1loss/SRqqt1/detection_loss‚0.5738965489402489 epoch‚14 0loss/FRt1/assignment_loss‚0.7606745424880285 2loss/FBt2/assignment_loss‚0.042120351892581606 /loss/FRt2/detection_loss‚0.4712621977687028 %loss/total_loss‚8.085487004327419{´‡Þˆ!‰ 0loss/FRt2/assignment_loss‚1.6095139884400642 /loss/FRt1/detection_loss‚1.3869279400067038 /loss/FBt2/detection_loss‚0.8677486477662688 _runtime‚6071.132760783 1loss/SRqqt2/detection_loss‚0.8025897804196134 /loss/FBt1/detection_loss‚0.4486955418517547 trainer/global_step‚11699 2loss/SRqqt1/assignment_loss‚0.5357670756215805 2loss/SRqqt2/assignment_loss‚0.5498417903210573 1loss/FBt1/assignment_loss‚0.03644859881081501 1loss/SRqqt1/detection_loss‚0.5738965489402489 epoch‚14 0loss/FRt1/assignment_loss‚0.7606745424880285 2loss/FBt2/assignment_loss‚0.042120351892581606 /loss/FRt2/detection_loss‚0.4712621977687028 %loss/total_loss‚8.085487004327419 % _timestamp‚1.7844925945580835e+09 _step‚949µÒc·ôh‰!‚(:Ý ¶äôÒ½€ùË disk.rbd0.out‚ 0.4697265625 disk.nvme1n1.in‚0 disk.md0.out‚ 157.18359375 disk.dm-0.in‚ 4.3359375# memory_percent‚95.4880952835083( proc.memory.percent‚17.3581600189209 cpu‚26.14754709944162) disk./.usagePercent‚76.32298471872902$ disk./.usageGB‚1171.748622894287 disk.dm-1.in‚0 disk.nvme0n1.in‚ 4.3359375# disk.dm-0.out‚62409.33544921875( proc.memory.availableMB‚ 739.23046875" proc.memory.rssMB‚ 2843.9609375' disk.nvme0c0n1.out‚51309.9541015625( disk.nvme1c1n1.out‚20547.72412109375 proc.cpu.threads‚17% disk.nvme0n1.out‚51309.9541015625 disk.nvme1c1n1.in‚0 network.recv‚ 1.629032e+06 disk.dm-1.out‚ 9110.24609375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375& disk.nvme1n1.out‚20547.72412109375 disk.md0.in‚0 network.sent‚ 1.6137705e+07 „9 ŸŠ!‚(:” ¶äôÒ—õ¬Ì gpu.0.gpu‚70.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚126.563' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚42.187666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#(ã_‹!‚(:Ô ÅäôÒ¤ÏË" disk.nvme0n1.out‚ 51310.9453125$ memory_percent‚96.99223041534424( proc.memory.availableMB‚ 492.79296875 network.recv‚ 1.633368e+06) disk./.usagePercent‚76.32297974924877 disk.md0.out‚157.625# disk.dm-0.out‚62409.93701171875 disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 network.sent‚ 1.6188129e+07 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 51310.9453125 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚20548.39501953125" proc.memory.rssMB‚ 2843.9609375( proc.memory.percent‚17.3581600189209 cpu‚26.14886667837512 disk.nvme1n1.in‚0 disk.dm-1.out‚ 9110.36328125 proc.cpu.threads‚17% disk./.usageGB‚1171.7485466003418& disk.nvme1n1.out‚20548.39501953125 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375ž†÷žŒ!‚(:“ ÅäôÒ£†Ì gpu.0.gpu‚50.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚251.941' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚83.98033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©Ýî2!- _wandbruntime‚6096  _runtime‚6096y°j‘ÉŽ!à ' lr-AdamW/pg1‚0.00017453569686932992 ' lr-AdamW/pg2‚0.00017453569686932992 trainer/global_step‚11749 $ _timestamp‚1.784492620273951e+09 _runtime‚6096.848901474 _step‚950à™•!Î!È ' lr-AdamW/pg1‚0.00017453569686932992 ' lr-AdamW/pg2‚0.00017453569686932992 trainer/global_step‚11749 $ _timestamp‚1.784492620273951e+09 _runtime‚6096.848901474 _step‚950¶Võò2!- _wandbruntime‚6096  _runtime‚6096 Žôö'‘!ðO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚11749 % _timestamp‚1.7844926202749534e+09 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31066632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11277415648256 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 210292297 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚951 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚714131216698880 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91528778 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚714131192274432 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11257472882778 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚899031161437696 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11623164291584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚900870818376458 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚900870461413776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11277756434432 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118763517 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31066913 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118763517 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118753374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚899031143919104 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11277415648256 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91528780 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91528778 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11623293267456 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62313542 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 _runtime‚6096.849947265 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚725754509966336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚899031143919104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210282152 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118753374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62313540 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚899031161437696 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11277756434432 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚725754356566016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚910308559567360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚910308559567360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91528778 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11623293267456 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62313542 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 _runtime‚6096.849947265 TDDeviceStatsMonitor.on_train_batch_start/active_bytes;]£Ü.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚725754509966336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚899031143919104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210282152 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118753374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62313540 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚899031161437696 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11277756434432 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚725754356566016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚910308559567360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚910308559567360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31066632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11277415648256 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 210292297 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚951 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚714131216698880 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91528778 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚714131192274432 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11257472882778 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚899031161437696 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935·ò‰äª2“!- _wandbruntime‚6097  _runtime‚60971í<ì'”!úM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚899105353206784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91536282 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118763249 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11278686352896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚899105353206784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31069473 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11278345559552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚910384039559680 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 _step‚952 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210309688 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚910384039559680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚899105335688192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62318621 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚900944815719492 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11278345559552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210309688 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93388096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 210299529 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93387806 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚725814343964672 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚910383681247744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62318623 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31069185 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11258401106126 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚725814201057280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚889686771576160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91536280 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11624132450304 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118763249 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 _runtime‚6097.224547659 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚900945172682286 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91536282 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118763249 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11278686352896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚899105353206784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31069473 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11278345559552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚910384039559680 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 _step‚952 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210309688 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚910384039559680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚899105335688192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62318621 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚900944815719492 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11278345559552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210309688 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93388096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 210299529 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93387806 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚725814343964672 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚910383681247744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62318623 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31069185 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11258401106126 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚725814201057280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚889686771576160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91536280 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11624132450304 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118763249 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 _runtime‚6097.224547659 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚900945172682286 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 F¡!ðO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119248976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31193713 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11323943470080 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚904588208602784 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7844926461980517e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚914064675843584 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚893284632578320 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚914064675843584 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚914065034148352 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚902740732373504 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93761305 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119259119 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119259119 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91903878 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7844926461980517e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚914064675843584 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚893284632578320 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚914064675843584 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚914065034148352 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚902740732373504 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93761305 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119259119 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119259119 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91903878 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11325214174720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚717133885234176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211180390 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚914140155835904 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11324873381376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚902814941661184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119269008 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚902814924142592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211170231 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91911380 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91911382 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93768562 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62572673 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚902814924142592 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31195891 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚728806164562944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚904662919871294 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 _runtime‚6123.148575025 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211170231 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119258851 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211180390 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚893358058660800 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 % _timestamp‚1.7844926465734432e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 145004544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11672134324224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91911382 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚717133909658624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚904662562908500 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375¸ë ø¥!ÿM P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91911382 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚717133909658624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚904662562908500 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11325214174720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚717133885234176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211180390 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚914140155835904 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11324873381376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚902814941661184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119269008 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚902814924142592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211170231 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91911380 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91911382 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93768562 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62572673 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚902814924142592 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31195891 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚728806164562944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚904662919871294 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 _runtime‚6123.148575025 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211170231 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119258851 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211180390 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚893358058660800 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 % _timestamp‚1.7844926465734432e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 145004544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11672134324224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360¼AŒå2¦!- _wandbruntime‚6123  _runtime‚6123Í%´ §!… 0loss/SRqqt2/detection_loss‚0.918052243422481 epoch‚14 2loss/SRqqt2/assignment_loss‚0.6259682006950497 3loss/FBt1/assignment_loss‚0.0008729274183798233 /loss/FRt2/detection_loss‚0.5473887658248591 _step‚957 3loss/FBt2/assignment_loss‚0.0010366148143627167 /loss/FBt1/detection_loss‚0.5252084416301491 % _timestamp‚1.7844926465745072e+09 _runtime‚6123.152410871 1loss/SRqqt1/detection_loss‚0.5913480352970604 /loss/FBt2/detection_loss‚0.7404348745942115 %loss/total_loss‚8.359778793617902 trainer/global_step‚11799 0loss/FRt1/assignment_loss‚0.8362317976698411 0loss/FRt2/assignment_loss‚1.6707509435140169 1loss/SRqqt1/assignment_loss‚0.501263741784503 /loss/FRt1/detection_loss‚1.4012222069529898ÚÛ½w¨!Š 0loss/SRqqt2/detection_loss‚0.918052243422481 epoch‚14 2loss/SRqqt2/assignment_loss‚0.6259682006950497 3loss/FBt1/assignment_loss‚0.0008729274183798233 /loss/FRt2/detection_loss‚0.5473887658248591 _step‚957 3loss/FBt2/assignment_loss‚0.0010366148143627167 /loss/FBt1/detection_loss‚0.5252084416301491 % _timestamp‚1.7844926465745072e+09 _runtime‚6123.152410871 1loss/SRqqt1/detection_loss‚0.5913480352970604 /loss/FBt2/detection_loss‚0.7404348745942115 %loss/total_loss‚8.359778793617902 trainer/global_step‚11799 0loss/FRt1/assignment_loss‚0.8362317976698411 0loss/FRt2/assignment_loss‚1.6707509435140169 1loss/SRqqt1/assignment_loss‚0.501263741784503 /loss/FRt1/detection_loss‚1.4012222069529898½¾÷ m©!‚(:â òäôÒÂüË disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚20551.0380859375 disk.md0.in‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 2843.9609375( proc.memory.percent‚17.3581600189209 network.recv‚ 1.644573e+06) disk./.usagePercent‚76.30953730521227' disk.nvme1c1n1.out‚20551.0380859375 disk.nvme1n1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1171.5421714782715 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625 disk.dm-0.in‚ 4.3359375( proc.memory.availableMB‚ 492.66015625 cpu‚26.150132220218214( disk.nvme0c0n1.out‚51323.12744140625 disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚51323.12744140625 disk.md0.out‚ 158.64453125" disk.dm-0.out‚62422.1884765625" disk.dm-1.out‚9110.71630859375$ memory_percent‚96.99304103851318 network.sent‚ 1.6271317e+07½Å’Yª!‚(:’ òäôÒ†ó®Ì gpu.0.gpu‚33.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚52.0 gpu.0.powerWatts‚117.815' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.27166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sž2«!- _wandbruntime‚6148  _runtime‚6148óQ©óȬ! trainer/global_step‚11849 % _timestamp‚1.7844926714684882e+09 _runtime‚ 6148.0434767 _step‚958 ' lr-AdamW/pg1‚0.00017387897674601442 ' lr-AdamW/pg2‚0.00017387897674601442þ‚“EÍ­!Ç % _timestamp‚1.7844926714684882e+09 _runtime‚ 6148.0434767 _step‚958 ' lr-AdamW/pg1‚0.00017387897674601442 ' lr-AdamW/pg2‚0.00017387897674601442 trainer/global_step‚11849¾A2®!- _wandbruntime‚6148  _runtime‚6148Šp‚Æö'¯!ðO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚906450320827904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 212023556 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚917821150424576 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚896955953741680 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31320066 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92278978 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11719334649856 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11350359012794 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11370812078080 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚917820792119808 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119754721 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 _step‚959 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92278978 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11719205673984 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11350019089472 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94141708 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62821640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119744578 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚917820792119808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚731738184603136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94141425 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚906450338346496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11370471291904 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚908306312754474 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚906450338346496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11370471291904 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚11849 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚906450320827904 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 _runtime‚6148.047440997ß/; û'°!õO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚906450338346496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128975872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11370471291904 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153400320 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚908306312754474 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚906450338346496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11370471291904 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚11849 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚906450320827904 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 _runtime‚6148.047440997 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚906450320827904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 212023556 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚917821150424576 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚896955953741680 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31320066 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92278978 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11719334649856 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11350359012794 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11370812078080 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚917820792119808 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119754721 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 _step‚959 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92278978 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11719205673984 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11350019089472 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94141708 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62821640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119744578 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚917820792119808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚731738184603136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94141425¿wüžb2±!- _wandbruntime‚6148  _runtime‚6148zÿJ4 ²!úM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119764610 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚11849 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚897029379824160 _runtime‚6148.416467251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.o»ñáÆân_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94148956 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚917896272112128 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119754453 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚720077701861376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚906524512596992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31322531 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚296 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚906524530115584 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94149254 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212051092 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚908380667060302 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚917896272112128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚906524512596992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212040933 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11720296823296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11720169951744 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92286480 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119764610 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚298 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚906524530115584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0Àˆ™i'³!ÿM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92286482 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚731798023109120 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92286480 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚917896272112128 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119754453 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚720077701861376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚906524512596992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31322531 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚296 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚906524530115584 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94149254 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212051092 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚908380667060302 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚917896272112128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚906524512596992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212040933 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11720296823296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11720169951744 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92286480 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119764610 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚298 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚906524530115584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119764610 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚11849 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚897029379824160 _runtime‚6148.416467251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94148956 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B2¾!- _wandbruntime‚6172  _runtime‚6172¯åý,õ'¿!ïO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92654080 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120250323 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94522186 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11396802077802 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92654078 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 trainer/global_step‚11899 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120240180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚910159909282304 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206016 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92654080 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚734729855945216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚900627257865680 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚921576908396032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63075692 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124781568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚910159926800896 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚912023702980800 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚921576908396032 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120240180 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63075690 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚910159909282304 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94522485 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚912024059943482 $ _timestamp‚1.784492696275776e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11767338570752 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚921577266700800 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚722962642156032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 212904403 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚734730005151232 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 212894258 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145Šh˜|À!ôO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 212904403 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚734730005151232 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 212894258 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92654080 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120250323 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94522186 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11396802077802 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92654078 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 trainer/global_step‚11899 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120240180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚910159909282304 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149206016 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92654080 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚734729855945216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚900627257865680 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚921576908396032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63075692 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124781568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚910159926800896 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚912023702980800 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚921576908396032 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120240180 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63075690 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚910159909282304 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94522485 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚912024059943482 $ _timestamp‚1.784492696275776e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11767338570752 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚921577266700800 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚722962642156032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675ÃVvÐê2Á!- _wandbruntime‚6173  _runtime‚6173¼Äµt'Â!úM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚723021542913024 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92661580 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212921794 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚921652030076416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 % _timestamp‚1.7844926966476943e+09 _runtime‚6173.222842164 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120250055 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚910234118569984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31449002 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚912098414249310 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚910234101051392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11417929025024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212911635 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120260212 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚900700683948160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63080771 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚734789844765184 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133163008 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚921652388388352 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚921652388388352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94529773 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92661580 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120250055 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11397390377716 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31449294 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212921794 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11417929025024 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚910234101051392 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11418269818368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 _step‚964 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120260212 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11418269818368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚734789687177728 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11768168689152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 trainer/global_step‚11899 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚900700666908800 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11397730301150 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚910234118569984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212911635 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493òª9Q'Ã!ÿM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212911635 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120260212 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚900700683948160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63080771 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚734789844765184 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133163008 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 11508246528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚921652388388352 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚921652388388352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94529773 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92661580 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120250055 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11397390377716 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31449294 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212921794 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11417929025024 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚910234101051392 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11418269818368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 _step‚964 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120260212 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11418269818368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚734789687177728 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11768168689152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 trainer/global_step‚11899 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚900700666908800 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11397730301150 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚910234118569984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212911635 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚723021542913024 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92661580 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212921794 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚921652030076416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 % _timestamp‚1.7844926966476943e+09 _runtime‚6173.222842164 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120250055 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚910234118569984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31449002 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157587456 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚912098414249310 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚910234101051392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11417929025024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303ÄÊï{52Ä!- _wandbruntime‚6173  _runtime‚6173 lW Å!‡ 1loss/SRqqt2/detection_loss‚0.7023823144596648 /loss/FBt2/detection_loss‚0.7464562195997971 % _timestamp‚1.7844926966487079e+09 0loss/FRt2/assignment_loss‚1.5818202051926742 2loss/SRqqt2/assignment_loss‚0.4964209838401785 _runtime‚6173.223434145 /loss/FRt2/detection_loss‚0.5422265288985783 epoch‚14 trainer/global_step‚11899 _step‚965 0loss/FRt1/assignment_loss‚0.8508092188835144 2loss/SRqqt1/assignment_loss‚0.5280744691676116 3loss/FBt1/assignment_loss‚0.0006209127094870079 /loss/FRt1/detection_loss‚1.5468230192642658 1loss/SRqqt1/detection_loss‚0.6431215065490223 /loss/FBt1/detection_loss‚0.3773786705877522 %loss/total_loss‚8.017646316678274 3loss/FBt2/assignment_loss‚0.00151226752572816050\ðÆ!Œ /loss/FRt2/detection_loss‚0.5422265288985783 epoch‚14 trainer/global_step‚11899 _step‚965 0loss/FRt1/assignment_loss‚0.8508092188835144 2loss/SRqqt1/assignment_loss‚0.5280744691676116 3loss/FBt1/assignment_loss‚0.0006209127094870079 /loss/FRt1/detection_loss‚1.5468230192642658 1loss/SRqqt1/detection_loss‚0.6431215065490223 /loss/FBt1/detection_loss‚0.3773786705877522 %loss/total_loss‚8.017646316678274 3loss/FBt2/assignment_loss‚0.0015122675257281605 1loss/SRqqt2/detection_loss‚0.7023823144596648 /loss/FBt2/detection_loss‚0.7464562195997971 % _timestamp‚1.7844926966487079e+09 0loss/FRt2/assignment_loss‚1.5818202051926742 2loss/SRqqt2/assignment_loss‚0.4964209838401785 _runtime‚6173.223434145Å\vD³fÇ!‚(:Û ŸåôÒ÷ûðË disk.nvme0c0n1.in‚ 4.3359375 disk.md0.out‚ 159.67578125 disk.dm-0.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375$ memory_percent‚96.99256420135498( proc.memory.percent‚17.3581600189209( disk.nvme0c0n1.out‚51353.53662109375% disk./.usageGB‚1173.5469589233398 disk.dm-1.in‚0" disk.dm-1.out‚9111.02587890625" proc.memory.rssMB‚ 2843.9609375 cpu‚26.147145256644897 proc.cpu.threads‚17 network.recv‚ 1.655651e+06& disk.nvme0n1.out‚51353.53662109375 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0# disk.nvme1n1.out‚20552.75390625 disk.md0.in‚0 disk.dm-0.out‚62451.75390625( proc.memory.availableMB‚ 492.73828125% disk.nvme1c1n1.out‚20552.75390625 network.sent‚ 1.6400299e+07( disk./.usagePercent‚76.4401210827774 disk.rbd0.in‚0 disk.rbd0.out‚ 0.4697265625 üƒžÈ!‚(:“ ŸåôÒîÌ gpu.0.gpu‚55.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚52.0 gpu.0.powerWatts‚118.181' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.39366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áA9°É!‚(:Ü ®åôÒ³ÎË" disk.dm-1.out‚9111.09619140625( disk.nvme0c0n1.out‚53263.13720703125& disk.nvme0n1.out‚53263.13720703125 disk.nvme1c1n1.in‚0 network.sent‚ 1.6451891e+07 network.recv‚ 1.660197e+06) disk./.usagePercent‚76.47398510889283% disk./.usageGB‚1174.0668563842773 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375 disk.md0.out‚ 160.2890625! disk.dm-0.out‚64361©ý‰ ·.947265625 disk.dm-1.in‚0 disk.rbd0.out‚ 0.4697265625' disk.nvme1c1n1.out‚20554.7236328125 disk.nvme1n1.in‚0% disk.nvme1n1.out‚20554.7236328125$ memory_percent‚96.99230194091797% proc.memory.availableMB‚ 492.78125" proc.memory.rssMB‚ 2843.9609375 disk.rbd0.in‚0 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375( proc.memory.percent‚17.3581600189209 cpu‚26.14614792048485 proc.cpu.threads‚17ßa{ŸÊ!‚(:” ®åôÒàÐ…Ì gpu.0.gpu‚92.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚52.0 gpu.0.powerWatts‚118.459' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.486333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QM:Š2Ë!- _wandbruntime‚6197  _runtime‚6197™ƒ3ýÉÌ!à _step‚966 ' lr-AdamW/pg1‚0.00017321793496685402 ' lr-AdamW/pg2‚0.00017321793496685402 trainer/global_step‚11949 $ _timestamp‚1.784492721382007e+09 _runtime‚6197.957788527­‚|ÎÍ!È ' lr-AdamW/pg1‚0.00017321793496685402 ' lr-AdamW/pg2‚0.00017321793496685402 trainer/global_step‚11949 $ _timestamp‚1.784492721382007e+09 _runtime‚6197.957788527 _step‚966ÆÍÔ‘U2Î!- _wandbruntime‚6197  _runtime‚6197¯Zçßõ'Ï!ïO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31573914 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚297 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚913869497736704 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11463867721728 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚915741807132490 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚351 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93029178 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚737721830740480 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚904298544950320 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚925333024672256 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚925333382977024 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120735782 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11815347532800 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11463867721728 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 213775105 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63329742 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93029178 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94903656 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11442905219488 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 $ _timestamp‚1.784492721382796e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120745925 trainer/global_step‚11949 _step‚967 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚913869497736704 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚725906483207680 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚6197.957854547 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11443245142810 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120745925 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚913869515255296 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94903357 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11815214362624 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93029180 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93029180 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93029180 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157594624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 11508246528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93029180 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93029178 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚737721830740480 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚904298544950320 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚925333024672256 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚925333382977024 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120735782 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11815347532800 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11463867721728 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 213775105 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63329742 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93029178 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94903656 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11442905219488 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 $ _timestamp‚1.784492721382796e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120745925 trainer/global_step‚11949 _step‚967 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚913869497736704 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚725906483207680 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚6197.957854547 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11443245142810 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120745925 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚913869515255296 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592ÇåɃš2Ñ!- _wandbruntime‚6198  _runtime‚61983)'Ò!úM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135260160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚725965335115776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159684608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11444173366158 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 213792496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚725965359540224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93036682 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11816307496448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7844927217529407e+09 _step‚968 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63334821 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31576381 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93036680 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 _runtime‚6198.328087414 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚904371971032800 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120755814 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚737781507352064 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 213782337 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚915816161438318 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11464797640192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚925408504664576 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚925408146352640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11816172236288 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚913943689505792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚925408504664576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚915815804475524 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚913943689505792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120745657 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120745657 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94910915 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 213792496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93036682 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31576094 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11443833442724 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚913943707024384 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0Ñ´AÓ!ÿM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚913943689505792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚925408504664576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚915815804475524 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚913943689º;ÃúÄ#505792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120745657 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120745657 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94910915 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2254652928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 213792496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93036682 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31576094 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11443833442724 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚913943707024384 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 135260160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚725965335115776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159684608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 356962794 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11444173366158 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 213792496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚725965359540224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93036682 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11816307496448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7844927217529407e+09 _step‚968 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63334821 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31576381 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93036680 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 _runtime‚6198.328087414 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚904371971032800 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120755814 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚737781507352064 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 213782337 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚915816161438318 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11464797640192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚925408504664576 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚925408146352640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11816172236288ÈyP(E2Ô!- _wandbruntime‚6198  _runtime‚6198¾il Õ!„ 0loss/FRt1/assignment_loss‚0.7286098827069728 2loss/FBt2/assignment_loss‚0.000966162441058901 /loss/FRt1/detection_loss‚1.4549534709436174 %loss/total_loss‚8.207854301814987 $ _timestamp‚1.784492721754605e+09 2loss/SRqqt2/assignment_loss‚0.6788908098120805 1loss/SRqqt1/detection_loss‚0.6080676827810457 1loss/SRqqt2/detection_loss‚0.8144581074350707 0loss/FBt2/detection_loss‚0.48936003479449186 epoch‚14 _runtime‚6198.329483627 _step‚969 0loss/FRt2/assignment_loss‚1.6981311278634956 1loss/SRqqt1/assignment_loss‚0.510404615638944 2loss/FBt1/assignment_loss‚0.057573728339488206 /loss/FRt2/detection_loss‚0.4635366996527978 /loss/FBt1/detection_loss‚0.7029019794059231 trainer/global_step‚11949HÎÖ!‰ trainer/global_step‚11949 0loss/FRt1/assignment_loss‚0.7286098827069728 2loss/FBt2/assignment_loss‚0.000966162441058901 /loss/FRt1/detection_loss‚1.4549534709436174 %loss/total_loss‚8.207854301814987 $ _timestamp‚1.784492721754605e+09 2loss/SRqqt2/assignment_loss‚0.6788908098120805 1loss/SRqqt1/detection_loss‚0.6080676827810457 1loss/SRqqt2/detection_loss‚0.8144581074350707 0loss/FBt2/detection_loss‚0.48936003479449186 epoch‚14 _runtime‚6198.329483627 _step‚969 0loss/FRt2/assignment_loss‚1.6981311278634956 1loss/SRqqt1/assignment_loss‚0.510404615638944 2loss/FBt1/assignment_loss‚0.057573728339488206 /loss/FRt2/detection_loss‚0.4635366996527978 /loss/FBt1/detection_loss‚0.7029019794059231É"×;l×!‚(:á ½åôÒÁ´Ë) disk./.usagePercent‚76.47399976885953$ disk./.usageGB‚1174.067081451416 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚20555.51123046875 disk.md0.in‚0 disk.md0.out‚ 160.76171875! disk.dm-1.out‚9111.2744140625 disk.nvme0n1.in‚ 4.3359375 network.sent‚ 1.6454914e+07& disk.nvme1n1.out‚20555.51123046875 disk.dm-1.in‚0( disk.nvme0c0n1.out‚53264.14501953125& disk.nvme0n1.out‚53264.14501953125$ memory_percent‚96.99294567108154( proc.memory.availableMB‚ 492.67578125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.3359375" proc.memory.rssMB‚ 2843.9609375( proc.memory.percent‚17.3581600189209 network.recv‚ 1.663633e+06 disk.nvme1n1.in‚0 disk.dm-0.in‚ 4.3359375! disk.dm-0.out‚64362.541015625 disk.rbd0.out‚ 0.4697265625 cpu‚26.145154774477884õ] ÑžØ!‚(:“ ½åôÒ Ä•Ì gpu.0.gpu‚52.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚55.0 gpu.0.powerWatts‚255.071' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.02366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L§â2Ù!- _wandbruntime‚6223  _runtime‚6223aتÉÚ!à ' lr-AdamW/pg2‚0.00017288580668044473 trainer/global_step‚11999 % _timestamp‚1.7844927464330678e+09 _runtime‚ 6223.00805131 _step‚970 ' lr-AdamW/pg1‚0.00017288580668044473ín^ÊÎÛ!È ' lr-AdamW/pg1‚0.00017288580668044473 ' lr-AdamW/pg2‚0.00017288580668044473 trainer/global_step‚11999 % _timestamp‚1.7844927464330678e+09 _runtime‚ 6223.00805131 _step‚970ÊÐ>IÇ2Ü!- _wandbruntime‚6223  _runtime‚6223ïÌÁsö'Ý!ðO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚929089499253248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93404280 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11489688207818 _runtime‚6223.012445281 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11510054757376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31700574 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚917579086191104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 356962682 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93404278 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214645807 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63583790 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚907969849074320 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 214635662 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63583792 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11489348284496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93404278 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95284364 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚740713605260288 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121231384 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚917579103709696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚929089140948480 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚907969866113680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚971 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11510395543552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚919459554321498 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121241527 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚917579103709696 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚929089140948480 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93404280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31700870 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚917579086191104 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11863305425408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93404278 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 184863232 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95284364 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚740713605260288 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121231384 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚917579103709696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚929089140948480 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚907969866113680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚971 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11510395543552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚919459554321498 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121241527 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 131073024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚917579103709696 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚929089140948480 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93404280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31700870 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17518592 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚917579086191104 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11863305425408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11511325462016 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚728909176167424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95291897 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 _runtime‚6223.380887894 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚929164620940800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93411782 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93411780 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63588873 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63588871 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11490276507732 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11510984668672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11510984668672 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚917653295478784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚929164262628864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11864266957824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 133163008 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11864133794816 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214653039 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 $ _timestamp‚1.784492746805752e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 214653039 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11511325462016 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121251416 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121241259 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95292187 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31703026 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31703314 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚11999 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93411782 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚917653295478784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚908043292196160 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚972 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93411780 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121241259 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚929164620940800 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚917653277960192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚917653277960192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚728909151742976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214663198 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚908043275156800 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11490616431166 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214663198 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚740773443125248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214653039 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 $ _timestamp‚1.784492746805752e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 214653039 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11511325462016 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121251416 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121241259 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95292187 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31703026 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31703314 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚11999 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93411782 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚917653295478784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚908043292196160 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚972 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93411780 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121241259 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚929164620940800 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚917653277960192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚917653277960192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚728909151742976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214663198 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚908043275156800 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11490616431166 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214663198 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚740773443125248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11511325462016 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚728909176167424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95291897 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14407434240 _runtime‚6223.380887894 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚929164620940800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93411782 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1446 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93411780 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63588873 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358311936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63588871 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11490276507732 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 2164776960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260Ìàß-2â!- _wandbruntime‚6223  _runtime‚6223Ò¨ž@ã! epoch‚14 0loss/FRt2/assignment_loss‚1.6707212729473229 2loss/FBt1/assignment_loss‚0.003595615398033769 /loss/FRt1/detection_loss‚1.4499489549235465 1loss/SRqqt1/detection_loss‚0.6239116599581471 %loss/total_loss‚8.216299214318749 _runtime‚6223.385408846 _step‚973 2loss/SRqqt2/assignment_loss‚0.7375462481277282 1loss/FBt2/assignment_loss‚0.07321880663126669 /loss/FRt2/detection_loss‚0.5771533166848557 0loss/SRqqt2/detection_loss‚0.934034828885148 /loss/FBt1/detection_loss‚0.4571522921578435 trainer/global_step‚11999 $ _timestamp‚1.784492746806874e+09 0loss/FRt1/assignment_loss‚0.8088368421117661 2loss/SRqqt1/assignment_loss‚0.5745537527660423 .loss/FBt2/detection_loss‚0.305625623727048A,[= ä!† %loss/total_loss‚8.216299214318749 _runtime‚6223.385408846 _step‚973 2loss/SRqqt2/assignment_loss‚0.7375462481277282 1loss/FBt2/assignment_loss‚0.07321880663126669 /loss/FRt2/detection_loss‚0.5771533166848557 0loss/SRqqt2/detection_loss‚0.934034828885148 /loss/FBt1/detection_loss‚0.4571522921578435 trainer/global_step‚11999 $ _timestamp‚1.784492746806874e+09 0loss/FRt1/assignment_loss‚0.8088368421117661 2loss/SRqqt1/assignment_loss‚0.5745537527660423 .loss/FBt2/detection_loss‚0.305625623727048 epoch‚14 0loss/FRt2/assignment_loss‚1.6707212729473229 2loss/FBt1/assignment_loss‚0.003595615398033769 /loss/FRt1/detection_loss‚1.4499489549235465 1loss/SRqqt1/detection_loss‚0.6239116599581471͆ñÓcå!‚(:Ø ÌåôÒðÐÙË network.sent‚ 1.6518405e+07 network.recv‚ 1.66768e+06 disk.md0.in‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 4.3359375! disk.nvme0n1.out‚ 53837.921875& disk.nvme1n1.out‚20857.00537109375$ memory_percent‚96.99466228485107 disk.md0.out‚ 161.10546875 disk.dm-0.in‚ 4.3359375 disk.nvme0n1.in‚ 4.3359375( disk.nvme1c1n1.out‚20857.00537109375" proc.memory.rssMB‚ 2843.9609375) disk./.usagePercent‚76.47096118016967% disk./.usageGB‚1174.0204315185547# disk.dm-0.out‚65236.99951171875 disk.rbd0.out‚ 0.4697265625# disk.nvme0c0n1.out‚ 53837.921875 cpu‚26.14412201859922 proc.cpu.threads‚17! disk.dm-1.out‚9111.3564453125 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0( proc.memory.availableMB‚ 492.39453125( proc.memory.percent‚17.3581600189209ƒM—„žæ!‚(:“ ÌåôÒ¿Í·Ì gpu.0.gpu‚90.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚256.264' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.42133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vä†zmç!‚(:â ÛåôÒþ²êË disk.nvme0c0n1.in‚ 4.3359375% disk.nvme0n1.out‚53838.6826171875 disk.nvme1n1.in‚0 disk.md0.out‚ 161.3671875$ memory_percent‚96.99366092681885# proc.memory.rssMB‚ 2844.28515625) proc.memory.percent‚17.36013889312744 disk.nvme0n1.in‚ 4.3359375% disk.nvme1n1.out‚20857.4560546875 disk.dm-0.in‚ 4.3359375# disk.dm-0.out‚65237.54638671875 network.recv‚ 1.671934e+06) disk./.usagePercent‚76.45547975839261 disk.rbd0.in‚0( proc.memory.availableMB‚ 492.55859375 cpu‚26.147053145322314 proc.cpu.threads‚17 network.sent‚ 1.6581069e+07' disk.nvme0c0n1.out‚53838.6826171875 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚20857.4560546875 disk.md0.in‚0 disk.dm-1.in‚0" disk.dm-1.out‚9111.46044921875 disk.rbd0.out‚ 0.4697265625% disk./.usageGB‚1173.7827529907227ãȰ›è!‚(: ÛåôÒžÛ°Ì gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚53.0 gpu.0.powerWatts‚257.96' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.98666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê.ô}2é!- _wandbruntime‚6249  _runtime‚6249d|<¬—ê!Ýz 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22899922728538513 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_4‚1 jet/accuracy_4_of_5‚0 jet/accuracy_5_of_5‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_3_of_3‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.038257043808698654 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7580660581588745 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.31256845593452454 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8"Purity/2FRt0SRqqt1FBt/event_purity‚0.236111119389534 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6402648687362671 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12010089308023453 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_3‚1 jet/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 _runtime‚6249.395760508 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20989644527435303 (particle/accuracy‚0.8515238165855408 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9975353479385376 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9984127283096313 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4650000035762787 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8903509378433228 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9990079998970032 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13626739382743835 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt2SRqqt0FBt/event_purity‚0.199977844953537 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1574682891368866 epoch‚14 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.036439090967178345 particle/accuracy_1_of_1‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8846153616905212 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04288219287991524 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19632336497306824 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2033178061246872 ,jet/accuracy_4_of_4‚0.014682539738714695 /particle/accuracy_4_of_4‚0.9417514801025391 jet/accuracy_3_of_5‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_2‚0.7855408191680908 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07500000298023224 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9592510461807251 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01863165572285652 )jet/accuracy_1_of_3‚0.924793541431427 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6417656540870667 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_4_of_5‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9844576716423035 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9922288060188293 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1071428582072258 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_2_of_4‚0.7764221429824829 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6433264017105103 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4850906729698181 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 _step‚974 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01755007542669773 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 % _timestamp‚1.7844927728203998e+09 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6379329562187195 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9253242015838623 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7440969944000244 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8551047444343567 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6698717474937439 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3305349349975586 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_3_of_3‚0.07991977035999298 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8047007918357849 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.031225431710481644 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19326213002204895 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8939489722251892 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1299334615468979 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7224240303039551 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9969940185546875 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5116117000579834 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2214038074016571 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18817146122455597 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7461697459220886 particle/accuracy_1_of_2‚1 5validation_average_jet_accuracy‚0.365180641412735 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.728909969329834 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7355936169624329 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9624999761581421 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9464285969734192 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9954100847244263 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6417656540870667 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6070156693458557 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 particle/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.48711922764778137 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6064616441726685 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9945475459098816 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7959851026535034 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.32514885^#9Ì2833081e-05 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07500000298023224 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 )loss/val_total_loss‚7.838026698440271 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22226962447166443 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20815928280353546 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7264607548713684 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 +jet/accuracy_2_of_2‚0.15231898427009583 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5649413466453552 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6435583829879761 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5130208730697632 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7900000214576721 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3672839403152466 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18045121431350708 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1071428582072258 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20839321613311768 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6460769176483154 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9911996722221375 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22155757248401642 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7900000214576721 *jet/accuracy_1_of_4‚0.9802066087722778 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 )jet/accuracy_1_of_1‚0.320178359746933 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6872588396072388 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.27371877431869507 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7620028257369995 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925780296325684 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10520833730697632 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04846436530351639 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9973546862602234 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7"Purity/0FRt1SRqqt*FBt/event_purity‚0.63856440782547 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37651553750038147 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6353762745857239 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19391639530658722 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49077606201171875 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.670661985874176 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8764851689338684 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.43153616786003113 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ,"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.375 +particle/specificity‚0.9095814824104309 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2733590602874756 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +particle/sensitivity‚0.6391907930374146 trainer/global_step‚12014 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.030120912939310074 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.706904947757721 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6401641964912415 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6776199340820312 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7270010709762573 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4435114562511444 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6469101905822754 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01755007542669773 *jet/accuracy_2_of_3‚0.5620570778846741 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9978324174880981 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_4‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9905116558074951 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1697048842906952 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7"Purity/*FRt0SRqqt1FBt/event_purity‚0.90483158826828 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9968819618225098 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5751935243606567 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.723323404788971 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17945876717567444 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14639705419540405 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7949833869934082 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17169474065303802 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.20074693858623505 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03617808595299721 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8551047444343567 +jet/accuracy_3_of_4‚0.09139148145914078 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5659722685813904 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6898599863052368 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7338613867759705 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.763394832611084 particle/accuracy_2_of_2‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6216639280319214 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.025376765057444572 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1868099421262741 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3812181055545807 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9994463324546814 'particle/f_score‚0.6489397883415222 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.877193033695221 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8681318759918213 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6788184642791748 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9996646642684937 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6538958549499512 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6616499423980713 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20815928280353546 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8800125122070312 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0300392247736454 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935I‚h=ë!âz 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49077606201171875 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.670661985874176 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8764851689338684 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.43153616786003113 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ,"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.375 +particle/specificity‚0.9095814824104309 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2733590602874756 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +particle/sensitivity‚0.6391907930374146 trainer/global_step‚12014 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.030120912939310074 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.706904947757721 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6401641964912415 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6776199340820312 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7270010709762573 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4435114562511444 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6469101905822754 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01755007542669773 *jet/accuracy_2_of_3‚0.5620570778846741 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9978324174880981 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_4‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9905116558074951 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1697048842906952 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7"Purity/*FRt0SRqqt1FBt/event_purity‚0.90483158826828 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9968819618225098 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5751935243606567 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.723323404788971 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17945876717567444 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14639705419540405 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7949833869934082 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17169474065303802 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.20074693858623505 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03617808595299721 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8551047444343567 +jet/accuracy_3_of_4‚0.09139148145914078 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5659722685813904 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6898599863052368 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7338613867759705 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.763394832611084 particle/accuracy_2_of_2‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6216639280319214 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.025376765057444572 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1868099421262741 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3812181055545807 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9994463324546814 'particle/f_score‚0.6489397883415222 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.877193033695221 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8681318759918213 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6788184642791748 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9996646642684937 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6538958549499512 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6616499423980713 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20815928280353546 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8800125122070312 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0300392247736454 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22899922728538513 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_4‚1 jet/accuracy_4_of_5‚0 jet/accuracy_5_of_5‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_3_of_3‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.038257043808698654 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7580660581588745 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.31256845593452454 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8"Purity/2FRt0SRqqt1FBt/event_purity‚0.236111119389534 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6402648687362671 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12010089308023453 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_3‚1 jet/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 _runtime‚6249.395760508 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20989644527435303 (particle/accuracy‚0.8515238165855408 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9975353479385376 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9984127283096313 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4650000035762787 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8903509378433228 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9990079998970032 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13626739382743835 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt2SRqqt0FBt/event_purity‚0.199977844953537 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1574682891368866 epoch‚14 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.036439090967178345 particle/accuracy_1_of_1‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8846153616905212 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04288219287991524 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19632336497306824 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2033178061246872 ,jet/accuracy_4_of_4‚0.014682539738714695 /particle/accuracy_4_of_4‚0.9417514801025391 jet/accuracy_3_of_5‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_2‚0.7855408191680908 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07500000298023224 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9592510461807251 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01863165572285652 )jet/accuracy_1_of_3‚0.924793541431427 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6417656540870667 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_4_of_5‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9844576716423035 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9922288060188293 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1071428582072258 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_2_of_4‚0.7764221429824829 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6433264017105103 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4850906729698181 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 _step‚974 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01755007542669773 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 % _timestamp‚1.7844927728203998e+09 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6379329562187195 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9253242015838623 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7440969944000244 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8551047444343567 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6698717474937439 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3305349349975586 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_3_of_3‚0.07991977035999298 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8047007918357849 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.031225431710481644 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19326213002204895 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8939489722251892 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1299334615468979 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7224240303039551 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9969940185546875 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5116117000579834 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2214038074016571 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18817146122455597 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7461697459220886 particle/accuracy_1_of_2‚1 5validation_average_jet_accuracy‚0.365180641412735 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.728909969329834 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7355936169624329 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9624999761581421 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9464285969734192 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9954100847244263 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6417656540870667 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6070156693458557 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 particle/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.48711922764778137 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6064616441726685 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9945475459098816 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7959851026535034 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07500000298023224 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 )loss/val_total_loss‚7.838026698440271 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22226962447166443 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20815928280353546 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7264607548713684 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 +jet/accuracy_2_of_2‚0.15231898427009583 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5649413466453552 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6435583829879761 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5130208730697632 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7900000214576721 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3672839403152466 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18045121431350708 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1071428582072258 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20839321613311768 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6460769176483154 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9911996722221375 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22155757248401642 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7900000214576721 *jet/accuracy_1_of_4‚0.9802066087722778 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 )jet/accuracy_1_of_1‚0.320178359746933 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6872588396072388 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.27371877431869507 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7620028257369995 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925780296325684 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10520833730697632 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04846436530351639 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9973546862602234 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7"Purity/0FRt1SRqqt*FBt/event_purity‚0.63856440782547 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37651553750038147 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6353762745857239 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19391639530658722 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1ÎØh/Ñhì!‚(:Ý êåôÒ´ðË disk.dm-1.in‚0" disk.dm-1.out‚9111.56201171875# proc.memory.rssMB‚ 2844.62890625* proc.memory.percent‚17.362236976623535 network.sent‚ 1.6585899e+07 disk.rbd0.out‚ 0.4697265625# memory_percent‚96.9933032989502' proc.memory.availableMB‚ 492.6171875 cpu‚26.1502501371989 proc.cpu.threads‚17% disk./.usageGB‚1173.7833442687988 disk.nvme0c0n1.in‚ 4.3359375 disk.nvme1c1n1.in‚0 disk.md0.out‚ 161.66015625 network.recv‚ 1.67569e+06) disk./.usagePercent‚76.45551827186445( disk.nvme0c0n1.out‚53855.10107421875& disk.nvme0n1.out‚53855.10107421875' disk.nvme1c1n1.out‚20860.5537109375% disk.nvme1n1.out‚20860.5537109375 disk.md0.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 4.3359375 disk.nvme1n1.in‚0 disk.dm-0.in‚ 4.3359375 disk.dm-0.out‚ 65256.3046875ÛƒÐJí!‚(:’ êåôÒðáÉÌ gpu.0.gpu‚24.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚52.0 gpu.0.powerWatts‚117.83' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.276666666666664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^áL]2î!- _wandbruntime‚6266  _runtime‚6266IHj™Êï!Ä % _timestamp‚1.7844927896360776e+09 _runtime‚6266.211636357 _step‚975 ' lr-AdamW/pg1‚0.00017255261388338244 ' lr-AdamW/pg2‚0.00017255261388338244 trainer/global_step‚12049E®"|Ïð!É _step‚975 ' lr-AdamW/pg1‚0.00017255261388338244 ' lr-AdamW/pg2‚0.00017255261388338244 trainer/global_step‚12049 % _timestamp‚1.7844927896360776e+09 _runtime‚6266.211636357ÏÆ‘~~2ñ!- _wandbruntime‚6266  _runtime‚6266µ"”»‹ò!ðO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11581439417856 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚913558241401520 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14407434240 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚925118505715129 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13862174720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64024320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 545259520 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11581098631680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚745322941033984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚733387002553856 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚934817122235392 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 215911093 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64024322 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚976 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11935962904576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚934816763930624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚733386978129408 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94020544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95951293 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚745323098628608 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11560281352969 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121900694 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121900694 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1446 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 215921238 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 GADeviceSt×yzók atsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚923235665298944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 538870784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 215921238 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚923235665298944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 12214494720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358304768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚733387002553856 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚934817122235392 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 215911093 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 538001087 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64024322 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚976 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11935962904576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 545259520 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚934816763930624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚733386978129408 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94020544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95951293 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚745323098628608 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 2254652928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚935 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 133170176 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11560281352969 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121900694 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121900694 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 11712753664ÐZÕ¡2ô!- _wandbruntime‚6266  _runtime‚6266<àÀ'õ!úM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95958794 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚913631650444640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94028046 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11582028542976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64029403 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163878912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64029401 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121910583 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11582369336320 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121910583 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11561549499639 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚733445854461952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94028046 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚923309874586624 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚934892243922944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚12049 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚923309874586624 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚745382932418048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚934891885611008 _runtime‚6266.579352022 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11561209576205 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 215938629 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11937053531648 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚923309857068032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31929674 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215938629 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 _step‚977 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 215928470 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚745382768539136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚913631667484000 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11582028542976$ä™+E&ö!ÿM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚913631667484000 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11582028542976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95958794 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚913631650444640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚675 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94028046 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11582028542976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64029403 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163878912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64029401 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121910583 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚675 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11582369336320 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121910583 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11561549499639 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 12214494720 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358311936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚493 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚733445854461952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14407434240 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94028046 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚923309874586624 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 11712753664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚934892243922944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚12049 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚923309874586624 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚745382932418048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚934891885611008 _runtime‚6266.579352022 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 12214494720 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11561209576205 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 215938629 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24424448 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11937053531648 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚923309857068032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31929674 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215938629 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13862174720 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚260 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 538870784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚837 _step‚977 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚935 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 545259520 E=DeviceStatsMonitor.on_train_batch_end/segmen‡¯4‚Àt.small_pool.peak‚260 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 215928470 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚745382768539136Ñ.Lë2÷!- _wandbruntime‚6266  _runtime‚6266£/h ø!† %loss/total_loss‚8.415053229877095 _runtime‚6266.579921824 _step‚978 2loss/SRqqt2/assignment_loss‚0.5004065688165616 /loss/FRt2/detection_loss‚0.5715442081208698 epoch‚15 trainer/global_step‚12049 % _timestamp‚1.7844927900052326e+09 4loss/FBt1/assignment_loss‚0.00015097574731481878 .loss/FRt1/detection_loss‚1.518902320203743 /loss/FBt1/detection_loss‚0.5449320442421595 0loss/FRt1/assignment_loss‚0.8348248612704535 2loss/FBt2/assignment_loss‚0.006869031359937278 1loss/SRqqt2/detection_loss‚0.8151392244128225 0loss/FRt2/assignment_loss‚1.7288099942000017 2loss/SRqqt1/assignment_loss‚0.5743658766027775 1loss/SRqqt1/detection_loss‚0.6016382131885445 /loss/FBt2/detection_loss‚0.7174699117119113ˆºÔŠù!‹ 2loss/SRqqt2/assignment_loss‚0.5004065688165616 /loss/FRt2/detection_loss‚0.5715442081208698 epoch‚15 trainer/global_step‚12049 % _timestamp‚1.7844927900052326e+09 4loss/FBt1/assignment_loss‚0.00015097574731481878 .loss/FRt1/detection_loss‚1.518902320203743 /loss/FBt1/detection_loss‚0.5449320442421595 0loss/FRt1/assignment_loss‚0.8348248612704535 2loss/FBt2/assignment_loss‚0.006869031359937278 1loss/SRqqt2/detection_loss‚0.8151392244128225 0loss/FRt2/assignment_loss‚1.7288099942000017 2loss/SRqqt1/assignment_loss‚0.5743658766027775 1loss/SRqqt1/detection_loss‚0.6016382131885445 /loss/FBt2/detection_loss‚0.7174699117119113 %loss/total_loss‚8.415053229877095 _runtime‚6266.579921824 _step‚978Ò·/[Àmú!‚(:â ùåôÒ° ÃË$ memory_percent‚96.99509143829346* proc.memory.percent‚17.362236976623535' disk.nvme1c1n1.out‚20860.9482421875 disk.nvme0c0n1.in‚ 4.3359375( disk.nvme0c0n1.out‚53856.09326171875( proc.memory.availableMB‚ 492.32421875 cpu‚26.149377366147828 proc.cpu.threads‚17 disk.nvme0n1.in‚ 4.3359375 disk.dm-0.in‚ 4.3359375" disk.dm-1.out‚9111.62451171875# proc.memory.rssMB‚ 2844.62890625 disk.dm-0.out‚65257.16796875) disk./.usagePercent‚76.45551578712433% disk./.usageGB‚1173.7833061218262& disk.nvme0n1.out‚53856.09326171875 disk.nvme1n1.in‚0% disk.nvme1n1.out‚20860.9482421875 disk.md0.in‚0 disk.md0.out‚ 161.8671875 network.sent‚ 1.6642671e+07 network.recv‚ 1.679324e+06 disk.nvme1c1n1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚ 0.46972656250îmO’û!‚(:‡ ùåôÒ«’Ì gpu.0.gpu‚33.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚54.0 gpu.0.powerWatts‚255.588' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚85.196 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0WŸ]hü!‚(:Ý ˆæôÒ¿ºìË( disk.nvme0c0n1.out‚53869.29248046875* proc.memory.percent‚17.362236976623535 disk.md0.out‚ 162.34375" disk.dm-1.out‚9111.78857421875$ memory_percent‚96.99368476867676' proc.memory.availableMB‚ 492.5546875# proc.memory.rssMB‚ 2844.62890625 proc.cpu.threads‚17 network.sent‚ 1.6705251e+07 network.recv‚ 1.683606e+06 disk.dm-0.out‚ 65271.8671875 cpu‚26.14854257029493) disk./.usagePercent‚76.45551702949437% disk./.usageGB‚1173.7833251953125 disk.nvme0n1.in‚ 4.3359375& disk.nvme0n1.out‚53869.29248046875 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚20863.6044921875 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme1n1.in‚0% disk.nvme1n1.out‚20863.6044921875 disk.md0.in‚0 disk.dm-0.in‚ 4.3359375 disk.rbd0.out‚ 0.4697265625 disk.nvme0c0n1.in‚ 4.33593753Ëœý!‚(:‘ ˆæôÒ´Ì gpu.0.gpu‚32.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚17.413101196289062* gpu.0.memoryAllocatedBytes‚ 14957740032 gpu.0.temp‚52.0 gpu.0.powerWatts‚118.15' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.38333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"–>c2þ!- _wandbruntime‚6290  _runtime‚6290 ©>Éÿ!à ' lr-AdamW/pg1‚0.00017221836191242608 ' lr-AdamW/pg2‚0.00017221836191242608 trainer/global_step‚12099 % _timestamp‚1.7844928143790703e+09 _runtime‚ 6290.95540777 _step‚979UCó0΀"È ' lr-AdamW/pg1‚0.00017221836191242608 ' lr-AdamW/pg2‚0.00017221836191242608 trainer/global_step‚12099 % _timestamp‚1.7844928143790703e+09 _runtime‚ 6290.95540777 _step‚979ÓR¼ã2"- _wandbruntime‚6290  _runtime‚6290ÚÀæö'‚"ðO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11627626453504 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 trainer/global_step‚12099 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11984040933888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96331533 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7844928143798234e+09 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚748314700423168 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216781795 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚938572880206848 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚736330794756608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94395644 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11607064341299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 216791940 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11606724417977 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11983905666560 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚736330819181056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96331248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚980 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94395642 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94395642 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚926945253753344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11627967239680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64278372 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64278370 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚928836609866819 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚928836252904137 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122386153 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11627626453504Aõ6û'ƒ"õO G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚675 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216781795 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚938572880206848 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚736330794756608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 339923322 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94395644 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚935 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚675 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11607064341299 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚260 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1446 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 216791940 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24424448 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11606724417977 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 340786176 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14407434240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 2164776960 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11983905666560 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 538870784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚675 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚736330819181056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚260 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96331248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚935 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚980 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94395642 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94395642 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚926945253753344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11627967239680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17518592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64278372 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64278370 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚928836609866819 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 12030241375 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚928836252904137 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 545259520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚935 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122386153 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358304768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 11712753664 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14407434240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚260 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11627626453504 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 12214494720 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚837 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13862174720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 340786176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13862174720 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11627626453504 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚493 trainer/global_step‚12099 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11984040933888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96331533 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7844928143798234e+09 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚748314700423168Ô„w.2„"- _wandbruntime‚6291  _runtime‚62916Üô`…"úM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚748374697172992 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94403146 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚260 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122406185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 12030241375 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13862174720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 545259520 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚938648360199168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 184863232 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 545259520 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚927019445522432 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122396028 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚748374535391232 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11984864302080 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚938648001887232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14407434240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13862174720 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚935 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 11712753664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94403144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161781760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚917302954568640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 339923434 trainer/global_step‚12099 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚736389671089152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 137357312 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11607992564647 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚675 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 216809331 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17518592 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 122406185 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11985001659392 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 538001087 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 340793344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚935 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚260 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚351 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 538870784 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17518592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 340793344 F